GitOps & Secrets

So you’ve struck out on the GitOps path… You’ve got your application manifests templated out in Helm, they’re stored on Github, you’ve got ArgoCD facing your repo and an ApplicationSet putting everything together. “Ahhhhh”, you recline, feet on your desk, “this is the sweet life.” But something enters your mind. Something feels off. Sure, your pull requests are building ephemeral preview builds to the cluster, but what about the secrets? Up to this point, you had excluded secrets from your deployment templates, simply conceding that once an application is built, someone still has to log on to the cluster console to configure the secrets before the application will run properly. You wouldn’t want those precious API keys out on display on GitHub, your database credentials and connection strings out there for all to see… you need these secrets off your repositories! “Damn it all to hell!” you say and press the big red button that returns everything to 2-hour long Jenkins builds.

But wait… there might be a way around this. Press that other big red button that undoes the first one.

SealedSecrets

We can used SealedSecrets to package secrets opaquely. SealedSecrets are a feature of Kubernetes - signed secret objects exclusive to a specific cluster. And if we want to package our secrets alongside our code for dynamic deployment, they do the job nicely.

In the two-element system made up of a cluster-side operator and a client side utility, a normal Secret.yaml is passed from your local machine to the cluster, which encrypts its contents, and returns to you a SealedSecret object file. This SealedSecret file is only readable by the cluster that signed it - not even you can read it. These SealedSecrets can then be safely stored in plain sight, on a public repository for example, safe in the knowledge that it can be used only by the intended cluster. This is the key for our initial problem - successfully obscuring a secret from local development, all the way through the pipeline, the secret only becoming available to the system intended to run it.

A Word on Key Management Software

In my opinion, considering their apparent simplicity, SealedSecrets accomplish a lot. If we’re defining our problem as how to safely package sensitive information for GitOps deployments, they provide an tidy way to do this. But we’re still handling credentials manually in the sealing process. If we want to look beyond this, to a future you can be more even more hands off with management of the secrets’ actual values, implementing or integrating an existing key management platform to your GitOps flow could be the solution. Together, the tools potentiate one another and provide not only a secure way of providing sensitive values to your Helm builds, but secure storage of the values, automatic provisioning and cycling of values where needed, reducing the need for manual handling of secure data. Key Management Software, version control, build tools - can all be considered as sources for these values.

The Why

There are numerous ways one could implement secret values in a GitOps workflow, but in these posts I’d like to concentrate on the why behind particular Dev&GitOps tools, and avoid too much technical detail. There are a few of these options I’d like to cover here in the future: