Poor Man's High Availability Vault Server With Auto Unseal Using KMS

Hashicorp's Vault provides a solution to secure, store and control access to secrets and other sensitive data. There are multiple ways to achieve a highly available Vault setup, some might use a cluster of vault servers, a few others might go with a master slave deployment strategy. In this blog we will be choosing a poor man's approach to achieve high availability for a non complex and small scale project with minimal configuration and resources.

This post assumes that you have a basic understanding of the Vault and how to install it. To know more about installing the Vault server on an EC2 instance, please refer to our blog post on SSH Access Management with Vault

Why Auto Unseal for HA?

Before we move into the details of implementation, we need to answer the question why Auto Unseal is necessary for high availability? Vault server always boots up in the sealed state. Data in the Vault storage is always stored in an encrypted format. Vault is configured to know how to access the physical storage and its location, but doesn't know how to decrypt any of it. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data. If the Vault server restarts itself or is manually restarted by an user, it goes back to the sealed state and will have to be manually unsealed. This can cause major outages which would come up as an unnecessary surprise during off hours.

Auto Unseal is a process where the keys required to unseal the Vault are stored in the key management system like AWS KMS and is made available to the Vault to unseal itself on accidental or forced reboot without any manual interruption required.

Components used in this solution:(We will be using AWS as cloud provider)

AWS KMS Key
AWS IAM policies and roles required to access AWS KMS Key
EC2 server with Auto Recovery (To minimize the outage caused due to hardware failure)
Vault binary to be deployed on EC2

IAM and KMS Setup:

Start the implementation by creating the IAM policy called VaultKMSUnsealPolicy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
      "Resource": "*"
    }
  ]
}

Create an IAM role called VaultKMSUnsealRole and attach the VaultKMSUnsealPolicy to it.
Create an EC2 instance enabled with Auto-Recovery. (To know more about Auto recovery go through this AWS Blog)
Now attach the VaultKMSUnsealRole role to the EC2 instance. (If you already have an IAM role attached to your instance, skip step 2 and attach the VaultKMSUnsealPolicy to the existing role directly)
Go to AWS KMS section and create a new synchronous key called vault. Make note of the key id, as we will be using this in the next step.

Vault Config Setup

Download and install the Vault binary.

Below step has to be performed before initialising the vault. Once initialised it would already have the 5 keys provided by default which has to be then used to unseal the vault manually.

Caution: This step has to be completed before vault operator init is run.

Modify or create the vault config /etc/vault.d/vault.hcl to include below module.

seal "awskms" {
  region     = "${aws_region}"
  kms_key_id = "${kms_key}"
}

awskms is the module used by Vault to communicate with the AWS KMS to fetch and update the keys
aws_region refers to the AWS region where the AWS KMS key is setup
kms_key_id is the key id of the AWS KMS key(vault) created in IAM and KMS Setup section

Once the config has been updated with above content, initialise the vault with below command.

$ export VAULT_ADDR="http://127.0.0.1:8200"
$ vault operator init >> ~/vault.keys
Initial Root Token: s.######

Unseal Key 1: key1###
Unseal Key 2: key2###
Unseal Key 3: key3###
Unseal Key 4: key4###
Unseal Key 5: key5###

Yes, we did it!!!

We now have a Vault server which is setup with Auto Unseal feature, deployed over an auto recoverable EC2 instance. Now if your EC2 instance reboots due to a hardware failure or by an accident caused by a human error the Vault server will automatically boot up in an unsealed state and will be ready for use without any manual intervention.