Access to EC2 instances isn’t the most sexy topic in days of serverless-first -architectures but reality is there are still valid use-cases for terminal access to VMs. The Question is how to do it in a modern way and keep your dev, ops and sec -teams happy.

Viewpoints

I did compare different options from the following viewpoints;

  • Connectivity; Is end to end network connection from client to host required or is out-of-band connection, through AWS API, used instead? Network connections implies correct configuration of VPC CIDR, routing, NACLs, security groups etc. There must also be internet gateway or vpn/direct connect attached to enable connection from outside of VPC.

  • Authentication; Is authentication based on linux users and ssh keys or does authentication use IAM identities? When users and keys are used, you should think how to manage them. Typical options are automation tools such as Ansible/Chef/Puppet or using AD/LDAP authentication. All of these are non-trivial to implement and operate at large scale.

  • Auditing; Is there centralized audit trail of (attempted) connections and can content of sessions be logged? As these logs can contain sensitive information is it essential to make sure logs are protected from un-authorized access. This could be e.g. storing the logs on separate AWS account.

Connectivity Options

  1. EC2 Instance Connect

    • Network connection to (bastion) host required.
    • Authentication based on AWS IAM identity and policies.
    • Connections logged to Cloudtrail, no session logging.


    EC2 Instance Connect is a way to distribute short-lived SSH keys and control access by IAM policies. Actual connection is created with SSH client and all normal requirements (network connectivity, user on host etc) apply. Host must also have ec2-instance-connect “agent” installed.

  2. SSM Session Manager

    • Out-of-band connection through AWS API.
    • Authentication based on AWS IAM identity and policies.
    • Connection and session logging to Cloudtrail and Cloudwatch Logs.


    SSM Session Manager provides terminal sessions to hosts with SSM agent via AWS API. There is cmd-line client and browser based access from AWS console. While no in-bound connections to host is required, SSM agent must be able to connect AWS SSM endpoint. You can do this using normal internet gateway or provision SSM endpoints into your VPC. This allows you to connect a host in VPC without any other public or private network access.

  3. SSM Session Manager with tunneling

    • Out-of-band connection through AWS API.
    • Authentication based on AWS IAM identity and policies (+ users and keys)
    • Connections logged to Cloudtrail, no session logging.


    Session Manager tunneling option allows you to use standard SSH client and copy files to host via AWS API. Naming is bit misleading as it doesn’t allow tunneling anything else but ssh and scp. It’s worth noticing that even when using standard ssh client, connection is made via AWS API and no network level connection to target host is required. Tunneling requirements and setup are slightly different from basic session manager.

    UPDATE: I wrote a follow-up to this topic, Tunneling into VPC, describing how to setup a generic VPN tunnel and connect to any resource inside the VPC.

  4. Traditional SSH

    • Network connection to (bastion) host required.
    • Authentication based on users and keys.
    • Build your own log management (or bring in a 3rd party solution)


    This is your normal SSH as you know it :-)

Conclusions

I would say the winner is SSM Session Manager (with or without tunneling). Having SSM agent present on host, not only enabled session manager but allows you do all kinds of sys-admin automation and reporting, that will result less and less need for terminal sessions.

Using Session Manager lets you to delete your bastion host and close SSH port on instances. Not having to maintain users and keys on hosts will not just simplify server management tasks but also pave the road towards serverless architecture and managing the services, not the servers.

Things to keep in mind when planning to use Session Manager

  • In your SSH host logs, tunneled connections are coming from *.compute.amazonaws.com, not from your clients address as you might expect. If your access control or intrusion detection is based on client address you may need to rethink that.

  • It is not possible to do port forwarding (other than ssh and scp) from you localhost. This can be see either as disadvantage or great feature depending on your priorities.

UPDATE: Since Aug 28, 2019 Session Manager supports generic port forwarding!