Recently, I configured Kubernetes secrets for a Rails app to be stored in git for deploying the app using the GitOps approach for one of our clients. This blog post is about the approach I followed to store the Kubernetes secrets for GitOps.

Storing Kubernetes manifest on git

We store the deployment, configmap, ingress & service YAML files in Gitlab. Hence some part of GitOps is already done. For the secrets.yml I had to first export secrets from the Kubernetes cluster to yaml since it's not stored anywhere else using:

kubectl get secrets my-app-secret -o yaml > secrets.yaml

This is how secrets.yaml content looks like:

apiVersion: v1
data:
  user: cGFzcw==
kind: Secret
metadata:
  name: my-app-secret
type: Opaque

Secrets format

In Kubernetes, secrets are store encoded in base64 format, which can be decoded easily. When you follow GitOps and secrets are to be stored on git, they can't be stored in base64 format, they need to be encrypted & stored on git. I came across kubeseal to solve this.

What is Kubeseal?

Its a tool used for encrypting Kubernetes secrets. Kubeseal will be used to make SealedSecrets as templates for secrets. A SealedSecret will be a CRD(Custom Resource Definition) that can be decrypted only by the kubeseal controller running on your Kubernetes cluster. The controller then creates a Kubernetes secret on the cluster.

Using kubeseal

You can install kubseal by following the instructions from its README.md.

The kubeseal consits of two parts.

1. Client - Installed locally.

2. Controller - Installed on the remote cluster.

For the controller to decrypt the SealedSecret it needs a certificate. While creating a SealedSecret locally we will use the certificate to encrypt it locally. Following is the command to fetch the certificate.

kubeseal --fetch-cert > staging.pem

Now with this certificate we can create a SealedSecret.

echo -n foo | kubeseal --raw --from-file secrets.yml --cert staging.pem -o yaml --name mysecret

which returned:

AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...

but we need a SealedSecret resource as yaml instead of just a secret. We need to pass a Kubernetes secret.yaml file for kubeseal to create a SealedSecret for us.

kubeseal --cert staging.pem --format=yaml < secrets.yml

which returns:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-app-secret
  namespace: default
spec:
  encryptedData:
    my-db-pass: AgChw/NSDxfhun7kWZZnXHR6zj...
  template:
    metadata:
      creationTimestamp: null
      name: my-app-secret
      namespace: default
    type: Opaque

Now I can output it to a yaml file as:

kubeseal --cert staging.pem --format=yaml < secrets.yml > sealedsecret.yml

The above steps are helpful when we don't have full access to the Kubernetes cluster for using the kubeseal controller directly.

Other ways to create SealedSecret

There is another way to generate a sealedsecret with kubeseal command, we need to pass secret.yaml to kubeseal command as input. Note that we are not passing the certificate manually this time, the controller will fetch it automatically.

kubectl create secret generic app-secret --dry-run --from-literal=foo=bar -o yaml | \
 kubeseal \
 --controller-name=kubeseal-sealed-secrets \
 --controller-namespace=kubeseal \
 --format yaml > sealedsecret.yaml

this will create a sealedsecret.yaml. The sealedsecret.yaml can be committed to git along with other kubernetes manifest files for GitOps. Using these manifest files we can deploy our app using any CD tool. In our case, we deployed using Argo CD, an opensource GitOps continuous delivery tool for Kubernetes.

When the sealedsecrets.yaml is deployed to the Kubernetes cluster the controller will unseal the SealedSecret automatically & create a secret. It will also watch for any changes to the SealedSecret & will update the corresponding secret.