SSH via bastion host with ForwardAgent

While it is pretty common to have an infrastructure behind load-balancers and bastion hosts, there are still many confusion around actual configuration of the SSH client for fast and convenient use of the setup. While I am not going to talk about actual advantages of bastion hosts, I will put here some clarifications on the SSH client setup.

Assuming you you a bastion.host that you user as a connection gateway to your private.host and you want to work with your default SSH key that is only on your local PC/laptop, you have two possible way.

The one and the most commonly used is with SSH Agent forwarding, meaning you have to run ssh-agent on you laptop, add the SSH keys to it via ssh-add command (or use ssh-add -L to list all keys in the agent) and then user ForwardAgent yes in ~/.ssh/config, something like this:

Host bastion.host
    User ssh_user
    HostName bastion.host
    ProxyCommand none
    IdentityFile ~/.ssh/id_rsa
    PasswordAuthentication no
    ForwardAgent yes
Host private.host
    User ssh_user
    ProxyCommand ssh -q -A bastion.host -W %h:%p
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent yes

And while this is all cool from one point of view, this method has few drawbacks:

• Running ForwardAgent is not a good idea in terms of security, and you can read more about it here.
• Running ForwardAgent requires you to actually configure and run ssh-agent on you local PC/laptop, which is not a big deal at all, but you will have to remember and check it all the time or you will have all kind of authentication errors and will spend sometime to find out the reason for them (not running/misconfigured agent

The second method to achieve the same functionality in terms of bastion host and avoid messing around ssh-agent is to use SSH ProxyCommand. In this scenario, when configured properly, ssh will first run the ProxyCommand to establish the connection to bastion.host and then enable the tunnel via this connection to the private.host. This way bastion.host will know nothing about your keys or anything related to authentication, but will just make a tunnel (similar to SSH port forwarding) and keep it for you until you are done.

To get this to work, you would adjust the ~/.ssh/config as follows:

Host bastion.host
    User ssh_user
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent no
Host private.host
    User ssh_user
    ProxyCommand ssh -W %h:%p -q bastion.host
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent no

So now as you have all in place and configured, you can ssh private.host and enjoy the stay on your secure server. While this is all cool, it has a lot of default things and assumptions behind the scene which you are not bothering to learn until you face a slightly different requirements: assume that you need to have the SSH configuration and per-host keys not in your home .ssh directory, but somewhere else. Lets say you have some /very/secure/location with separate ssh.conf (with the content from above) and a bastion.id_rsa and a private.id_rsa to use for the connections. To make them work you would assume that you only need to adjust the IdentityFile configurations to point to the correct keys and then run your SSH as follows: ssh -F /very/secure/location/ssh.conf private.host

Bad news – it will not work and will give you authentication error. Though you still will be able to access bastion.host with the above, you won’t be able to reach you final destination at private.host.

Good news – thanks to this lovely discussion at stack overflow, a minor adjustments have to be done in your ProxyCommand: you need to specify the ssh config file to it as well, so now it will look like: ProxyCommand ssh -F /very/secure/location/ssh.conf -W %h:%p -q bastion.host

Obviously the reason is that giving -F to initial ssh command, you instruct it to look for a specific configuration file, but when it will run the ProxyCommand, that instance of ssh client will have no clue whatsoever about your custom config and will look for default one in ~/.ssh/config and system-wide settings.

I’ve spent quite some time before I figured out what’s going on and in order not to do so again and hopefully save some of your time, let this post be here for future references