AWS Systems Manager Sessions Manager + EC2 Instance Connect = Awesome

AWS provides several ways to gain shell access to running ec2 instances. These include plain old SSH. You just need to start the instance with a public key for which you have the corresponding private key. This works fine for adhoc testing, prototyping, etc. But when it comes time to put workloads into production on EC2 SSH keys don't scale well. You should have one ssh key per system admin, since sharing credentials is a bad security practice. In addition to one key per person, you'll need a network path from the sys admin to the ec2 instance (public IP address on the EC2 instance, with IP restrictions on incoming connection, VPN connection to the VPC, etc). Also, you'll need to manage the SSH keys on the instances as your employees and contractors come and go.

Not ideal! Fortunately, there are some AWS services that can come to your rescue...

AWS Systems Manager Session Manager

AWS provides a service, Systems Manager Session Manager which, using an agent running on the EC2 instance and an IAM role attached to the instances, allows shell access to the instance without a direct network path between the operator and the instance or SSH running on the instance. The Session Manager backend in AWS land relays messages between the operator and the instance. The instance just needs access to the Session Manager api via public internet, NAT gateway, VPC interface, etc. Session Manager is a great solution to many of the problems with traditional SSH access since you no longer need to manage SSH keys on the instances since authentication is handled through IAM. You can manage shell access to the instances just like you manage your other IAM permissions.

Unfortunately Session Manager is not a complete drop-in replacement for traditional SSH. In addition to allowing shell access, Session Manager provides port-forwarding to ports on the instance you are connecting to; however, it does not provide any other port forwarding features like SSH does. A common use case for SSH port forwarding (that I’ve used myself) is connecting to a relational database that is not directly accessible from my development machine. Using a command like

ssh ec2-user@52.100.100.100 -i my-private-key.rsa -L

localhost:3306:some-database-url:3306

I can then use my favorite local MySQL client to connect to the database without any problems.

Now you can use SSM to port forward to port 22 on the EC2 instance and then connect via SSH over the SSM connection, but you will still need to manage the keypair for the instance(s). See documentation here.

AWS provides the following lines you can add to your ~/.ssh/config file to facilitate easily connecting to SSH over a SSM port forwarding.

host i-* mi-*

ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

then you can just use the following to connect to an ec2 instance (note you need the SSM plugin for the AWS CLI).

ssh ec2-user@i-someinstanceid -i yourpublickey.pub

AWS EC2 Instance Connection

Another method AWS provides for gaining shell access to an EC2 instance is via EC2 Instance Connect. EC2 Instance Connect does two things. First, it can provide a terminal experience in your browser for connecting to EC2 instances. In this case the AWS EC2 Instance Connect backend proxies your connection to the instance. For browser based access to work the EC2 instance must allow incoming connections from the EC2 Instance Connect IP addresses1. Second (and most importantly), EC2 Instance Connect provides the EC2 instance a temporary public key on-demand via the EC2 instance metadata service. This allows a user to connect using any keypair. The way this works is the SSH daemon is set up with a helper script that will check the EC2 instance metadata for public keys when authenticating incoming connections. The EC2 Instance Connect api call send-ssh-public-key provides your public key to the instance for 60 seconds. The keypair is only used to establish the connection so sessions are not limited to 60 seconds.

Like traditional SSH using EC2 Instance Connect requires a network connection from your local machine to the EC2 instance (or from the EC2 Instance Connect public IPs to the EC2 instance).

In my opinion the best part of EC2 Instance Connect is the ability to manage SSH access to your instances via IAM policies, instead of adding and removing public keys from the instances. Unfortunately you still need a direct networking connection from your local machine to the instance (public IP, VPN, etc).

Combine the two

By combining Session Manager and EC2 Instance Connect we can both connect without a direct networking connection from a local machine and get the full use of SSH without needing to manage SSH keys. How? Here are the steps

1. Upload the public key with the EC2 Instance Connect API
2. Use Session Manager to open a port-forwarded connection to port 22 on the instance
3. Connect via SSH using the keypair you provided to EC2 Instance Connect over the Session Manager connection

Note: all of these examples assume your region is set via a environment variable, and a public key is located at ~/.ssh/id_rsa.pub

First upload a public key

aws ec2-instance-connect send-ssh-public-key --instance-id

i-00123456789abcdef --availability-zone us-east-1a --instance-os-user

ec2-user --ssh-public-key file://~/.ssh/id_rsa.pub

Then SSH over SSM session port forwarding (using the ssh proxy command from above)

ssh ec2-user@i-00123456789abcdef

We can simplify this process using the following proxy command added to your ssh config (choose a different public key if you want).

host i-*

ProxyCommand sh -c "aws ec2-instance-connect send-ssh-public-key

--instance-id %h --availability-zone $(aws ec2 describe-instances

--instance-id %h --output text --query Reservations[].Instances[].Placement.AvailabilityZone) --instance-os-user ec2-user --ssh-public-key file://~/.ssh/id_rsa.pub && aws ssm

start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Explanation of the script: First the instance's availability zone is queried, then the public key is uploaded. Finally a connection is made to the instance via sessions manager with port forwarding for SSH.

Note that to use this you will need

1. A running instance with the SSM agent installed (latest Amazon Linux 2 and Ubuntu both have this by default)
2. The instance IAM role with necessary permissions for SSM agent (Use the AmazonSSMManagedInstanceCore managed policy)
3. EC2 Instance Connect setup on the instance (latest Amazon Linux 2 and Ubuntu both have this by default)
4. Your IAM user/role must have the necessary IAM permissions.2
5. The EC2 instance must be able to connect to the AWS Systems Manager API (via internet gateway, NAT gateway, or VPC endpoint, etc).

This combination of services has worked very well for us and is our preferred solution for EC2 shell access. Please try it out and let us know what you think or if you’d like to talk about how Trek10 can help put a solution like this in place for you.

Footnotes

1. EC2 Instance connect IP addresses as far as I know are only published in this list. Filter by region and service = EC2_INSTANCE_CONNECT.

* Allow ec2-instance-connect:SendSSHPublicKey on the instance’s arn
* Allow ssm:*Session on the instance’s arn and the AWS-StartSSHSession document arn.