Setup Bastion Server with SSH helper

Let us start by setting up an EC2 instance as the bastion host. We will be using Hashicorp's Vault SSH Helper to authenticate the users with the Vault server and provide access to users post confirmation from the Vault.

Vault SSH Helper is an agent used to communicate with the HashiCorp Vault's Server. It allows a machine to consume the One-Time-Password (OTP) created by our Vault server and then use it as client authentication credential for every SSH connection request. This helper agent must be installed on the target host that you want to SSH into, in this case, the bastion.

Install Vault SSH helper

# Download the vault-ssh-helper
$ wget https://releases.hashicorp.com/vault-ssh-helper/0.1.4/vault-ssh-helper_0.1.4_linux_amd64.zip

# Unzip the vault-ssh-helper in /usr/local/bin
$ sudo unzip -q vault-ssh-helper_0.1.4_linux_amd64.zip -d /usr/local/bin

# Make sure that vault-ssh-helper is executable
$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper

# Set the usr and group of vault-ssh-helper to root
$ sudo chown root:root /usr/local/bin/vault-ssh-helper

# Setup vault ssh-helper config /etc/vault-ssh-helper.d/config.hcl
sudo mkdir /etc/vault-ssh-helper.d
sudo touch /etc/vault-ssh-helper.d/config.hcl

#Contents of config.hcl has to be as below
vault_addr = "${VAULT_ADDRESS}"
ssh_mount_point = "ssh"
ca_cert = "-dev"
tls_skip_verify = true
allowed_roles = "*"

Replace ${VAULT_ADDRESS} with something like an ip address(http://ip.ip.ip.ip/) or domain(https://vault.example.com)
Value of 'ca_cert' can be set to '-dev' and 'tls_skip_verify' to 'true' if you want to use Vault over HTTP or are using it to test the setup before production(It is recommended to use Vault over HTTPS in production). This file should contain the certificate deployed for the Vault server and tls_skip_verify should be set to false. You can use the Amazon Certificate Manager to get certificate from AWS or get a Let's Encrypt certificate.

Setting up the pam.d config

# Create a backup of pam.d file
$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig

Modify the file /etc/pam.d/sshd to comment the standard UNIX authentication(@include common-auth) module and add vault ssh-helper modules as below

#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/tmp/vaultssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

Update the sshd service to pick the latest pam.d config

# Make a backup of the original
$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig

Add or modify the file /etc/ssh/ssh_config to include below changes:

ChallengeResponseAuthentication yes
PasswordAuthentication no
UsePAM yes

Restart the sshd service

sudo systemctl restart sshd

Create a Unix user which will be used to access the bastion server with OTP as password

# Creates linux user unix-user-name
$ useradd -ms /bin/bash unix-user-name
$ usermod -aG sudo unix-user-name

Logs for the vault-ssh-helper can be found below for debugging purposes

tail -f 100 /tmp/vaultssh.log

Setup Vault Client For User Access

This journey has been really long and we are now heading to the last stop. Every client or user who has to be authenticated by Vault and wants to SSH into the bastion server needs to install Vault client.

Install Client

Download vault client from link
Copy the executable to /usr/local/bin
Check if Vault is installed by checking the version

$ vault --version
Vault v1.3.1

Connect To The Vault Server

# Setup your Vault server address
$ export VAULT_ADDR="http://vault.example.com"

# You can create users through Vault UI and the login to vault with username and password to get the token
$ export VAULT_TOKEN=`vault login -token-only -method=userpass username=johndoe password=unthinkable`

# Or to get a quick preview you can export the vault root token generated on vault initialisation (Not recommended for production)
$ export VAULT_TOKEN="s.######"

# Check if the values have been set in the env
$ env | grep 'VAULT'
VAULT_ADDR=http://vault.example.com
VAULT_TOKEN=s.yourvaulttokenfornow

Get OTP From Vault Server

# username in the below result is the unix user we had created inside the bastion server. We are using the Vault OTP engine we created in part 1 of the blog series.
$ vault write ssh/creds/otp-engine ip=#{bastion-ip}
Key                Value
---                -----
lease_id           ssh/creds/otp-engine/DFjbgD5cN6dmwPuJ01SKzn9z
lease_duration     1h
lease_renewable    false
ip                 {bastion-ip}
key                99375f15-53d8-3388-7d62-0379a7e2034b
key_type           otp
port               22
username           unix-user-name

# Once you have obtained the OTP(key) you can now SSH into the bastion with unix-user and the OTP as password.
$ ssh unix-user-name@#{bastion-ip}
password:99375f15-53d8-3388-7d62-0379a7e2034b

The expiry time for the OTP can be customised by the administrator, it is recommended to set the expiry time under 30 minutes. Also the key or OTP obtained can only be used once. New request has to be made to obtain a new OTP in order to login again.

Conclusion

In this two part blog series, we learned how to handle SSH Access Management problem with a system designed using Hashicorp's Vault and the Bastion server.

Vault is known to be a Swiss Army Knife packaged with multiple functionalities, if you are interested then you can read more about the Vault on Hashicorp's Site.

SSH Access Management - Control SSH Access With Vault OTP Engine (PART 2)

Setup Bastion Server with SSH helper

Setup Vault Client For User Access

Conclusion