Managing Secrets at Scale: Why We Chose SOPS for Terraform and Kubernetes Secrets

At my recent company, I was part of a team managing over 10+ Kubernetes clusters, ensuring high availability and efficient deployment across multiple regions.

Our infrastructure was managed by tools like Terraform, ArgoCD, Atlantis and Backstage.

While I’ll dive deeper into this setup in a future post, today, I want to focus on how we securely managed our secrets — at scale.

---

I remember when we were still in the startup-ish phase—we already had a pretty good idea of how our infrastructure should be structured.

We liked to think of it in two layers:

• Infrastructure layer: Managed with Terraform and Atlantis to configure resources like IAM, networking, CloudSQL, GKE, and more.

• Application layer: Basically Kubernetes, with deployments handled by ArgoCD and Kustomize.

So, when we thought about managing secrets, ideally we’d only have to use a single tool to manage secrets in both layers.

From SSH keys and Kubernetes Secrets to GitHub tokens and API credentials, keeping these secrets secure was vital to our product’s security and compliance.

So we looked into several solutions like HashiCorp Vault, SealedSecrets and GCP Secret Manager but none fully met our needs:

• HashiCorp Vault:

  • We didn’t want to take on the operational burden and cost of managing it ourselves.

  • We were also hesitant to store secrets in their hosted solution, as it could lock us into their ecosystem and potentially require a paid license down the line.

• SealedSecrets:

  • Managing secret updates across clusters is complex. Not to mention, this is Kubernetes only solution.

  • Required SealedSecrets controllers in every cluster, adding operational overhead.

• GCP Secret Manager:

  • While convenient, as we already used GCP services, we were looking for a cloud-agnostic approach whenever possible

  • We would also need to pay for it

So this was a bummer.

But luckily, we found SOPS and we really liked it.

---

The SOPS repository README.md already provides a solid overview of its capabilities, so instead of repeating that, I want to share why we adopted it.

In short, it’s a tool to encrypt secret, be it in the YAML, JSON or ENV format.

Since we followed a GitOps approach and used GitHub as our single source of truth, one of the biggest advantages was that it allowed us to push encrypted secrets to our code repositories.

One must understand that even with private repositories, security isn't guaranteed—code leaks have happened before.

Just Google “GitHub Security Breach,” and you’ll see for yourself.

With SOPS, that risk was no longer a concern.

Unlike HashiCorp Vault, which requires maintaining a dedicated secrets storage, SOPS allowed us to store encrypted secrets directly in GitHub, keeping everything close to the code.

We encrypted everything—from secrets used by Terraform to Kubernetes Secrets deployed via ArgoCD.

To give you a little more context, we used the SOPS provider with Terraform, while ArgoCD synced with Kustomize files, automatically decrypting SOPS-encrypted secrets using ksops.

Another major benefit was how seamlessly SOPS integrated with a GCP KMS key.

• If you’re authenticated via gcloud CLI, SOPS automatically detects it and retrieves the KMS key when encrypting or decrypting secrets.

• We could enforce strict access policies on who could use these KMS keys, limiting who could create and encrypt new secrets.

💡 Check out the VSCode SOPS plugin—it automatically decrypts secrets when you view them and re-encrypts them when you make edits.

Now, let’s look at a simple example to see SOPS in action.

Kubernetes Secrets Example

In this example, we’ll encrypt a Kubernetes Secret manifest using GCP KMS and SOPS.

First, a quick rundown of the terms and tools, in case you ain’t familiar with it yet:

• Kubernetes Secret Object: A manifest holding secrets for Kubernetes use that keeps your confidential data separate from your application code.

• GCP KMS: A key management service for creating and managing encryption keys. Perfect for encrypting secrets in your Kubernetes cluster deployment.

Now, imagine you’ve got this Kubernetes Secret you want to store on your code repository:

To encrypt it, we’re gonna need SOPS — check out their documentation on how to install it.

Next, on the cloud provider side, like GCP in our case, create a GCP KMS key.

Ensure your GCP Account has the right permissions to access it.

To utilize GCP KMS key as a SOPS encryption key, create the following SOPS configuration file and place it at the root of your repository:

💡 encrypted_regex parameter will search for this parameter inside your file and encyrpt it.

Now we can encrypt our secret file using:

💡 My personal preference is to add .enc suffix to encrypted files.

That’s it — managing secret really shouldn’t be that hard.

---

⏪ Did you miss the previous issues? I'm sure you wouldn't, but JUST in case:

How We Manage On-Prem Infrastructure Access at Scale with Terraform and Teleport

---

I hope you find this resource helpful. Keep an eye out for more Cloud updates and developments in next week's newsletter.

Until then, keep 🧑‍💻-ing!

Warm regards, Teodor