Secrets in Code with Mozilla SOPS

Varokas Panusuwan

• 5 min read
Secrets in Code with Mozilla SOPS

Secrets presents a challenging dilemma for infrasture-as-a-code. Solution today converge mostly on storing these secrets in some external trusted system (Kube Secrets, Docker Secrets, Build System Secrest, Vault) outside of the code.

Using SOPS, we can check in the encrypted secrets (e.g. connection passwords) along with the code. The only thing out of sight is the encryption key wrapping these passwords.

The added value also is that SOPS understand structured file (JSON,YAML) and encrypt only values, leaving the keys intact to easily inspect.

Example – This file could be checked into github along with the code.

{
  "postgres_password": "ENC[AES256_GCM,data:PsyN..,type:str]",
  "aws_access_key": "ENC[AES256_GCM,data:PsyN..,type:str]",
}
mozilla/sops
Simple and flexible tool for managing secrets. Contribute to mozilla/sops development by creating an account on GitHub.
GitHubmozilla

Installation

MacOS 

$ brew install sops

Windows

  1. Go to the latest release page: https://github.com/mozilla/sops/releases/latest
  2. Download sops-v3.5.0.exe (or whatever the latest version is)
  3. Rename the file from sops-v3.5.0.exe to just sops.exe
  4. Copy the file sops.exe to C:\Windows\System32.
  5. Or – Alternately, put the file in any directory and set the Path environment variable accordingly

Encryption Key Configuration

The most important configuration for SOPS is what encryption key to use. In many case this is the only configuration needed.

Create a .sops.yaml at the root directory of the project with one of the configuration outlined below.

The key could be any PGP key. Alternately the key could also be provided by a Key Management Service (KMS) in AWS or Google Cloud. A good choice especially if the code eventually would be deployed to these cloud providers.

AWS KMS

A master key can be created in hardware security modules via AWS Key Management Service (KMS). We need arn of the key with a corresponding AWS Profile that has a permission to use the key.

$ aws --profile myprofile configure
AWS Access Key ID [None]: KEY_ID
AWS Secret Access Key [None]: SECRET_ACCESS_KEY

# Create .sops.yaml At project root
$ vi .sops.yaml
creation_rules:
  # If assuming roles for another account use "arn+role_arn".
  # See Advanced usage
  - kms: "arn:aws:kms:..."
    aws_profile: myprofile

The permission needed for key operations are:

"Action": [
  "kms:Encrypt",
  "kms:Decrypt",
  "kms:ReEncrypt*",
  "kms:GenerateDataKey*",
  "kms:DescribeKey"
],

For slightly more advanced use cases. We could also access master key from another account using AWS AssumeRole mechanism. This is particularly useful when we have one master key in production, but wanted to also access it from our staging AWS account. See https://github.com/mozilla/sops#assuming-roles-and-using-kms-in-various-aws-accounts

PGP (optionally via Keybase)

For personal use cases, using a PGP key probably suffice. If you are a keybase user, you already have a PGP key. We needed to do the followings

  1. Install GPG
  2. Export private keys from Keybase
  3. Import the keys to local machine
  4. (optional) Remove passphrase from the key
  5. Create .sops.yaml at the project
$ brew install gpg ## or apt install gpg 
$ gpg --import private_key.asc

... (take note of the key ID: 0701C740FB8D24E9)
...
gpg: key 0701C740FB8D24E9: secret key imported
...
...

# This is important for the passphrase screen to show up in console
$ export GPG_TTY=$(tty)
$ gpg --edit-key 0701C740FB8D24E9
gpg> passwd
# 1. Type current passphrase
# 2. Type "" (Blank)
# 3. Type "" (Confirm Blank)

$ vi .sops.yaml
creation_rules:
  - pgp: 0701C740FB8D24E9

Generate new GPG Key (And export)

Sometimes we want to generate and managed the GPG key without keybase

$ gpg --full-generate-key

Kind of key : (1) RSA and RSA (default)
keysize: 3072
Key is valid for? : 0 (Do not expire)

Real name: <name>
Email: <email>
Comment: (blank)

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

Enter Passphrase 2 times (cannot be blank)

public and secret key created and signed.

pub   rsa3072 2020-11-26 [SC]
      3F73E4821B848420CDEAEAD585E2DE2374D6377A  ---> Note this value
uid                      test1 <[email protected]>
sub   rsa3072 2020-11-26 [E]

# Remove the passphrase if wanted
# This is important for the passphrase screen to show up in console
$ export GPG_TTY=$(tty)
$ gpg --edit-key 3F73E4821B848420CDEAEAD585E2DE2374D6377A
gpg> passwd
# 1. Type current passphrase
# 2. Type "" (Blank)
# 3. Type "" (Confirm Blank)

Encrypt / Decrypt files

Create

Create a new file via sops will launch an editor

$ sops secret.enc.json
{
  "example_key": "example_value",
}

The resulting json would retain the same keys, with values encrypted. An extra metadata is added as an extra key in JSON

$ cat secret.enc.json                                                                                            
{
	"example_key": "ENC[AES256_GCM,data:PsyNr6jRJLIPN3P0tA==,iv:Ne63tk8f6uD9GLiHQoyrS/BrK4WL2I6+9Ul8nO6PkDw=,tag:rF8Hm4gm0+xlA3BqKivY7w==,type:str]",
	"sops": {
		...
        ... (metadata here)
	}
}%

Edit

Editing an existing files would launch and editor and encrypt the file.

$ sops secret.enc.json

Encrypt / Decrypt Existing Files

$ sops -e secret.json > secret.sops.json

$ sops -d secret.sops.json > secret.sops

⚠️ Important - If the PGP key has passphrase, make sure this environment variables is set or you will run into problems 

GPG_TTY=$(tty)
Cannot decrypt with GPG 2.2.5 and SOPS 3.0.0 · Issue #304 · mozilla/sops
It appears the utility is looking for a secret key in a file but my GPG installation (through macOS homebrew) uses the gpg-agent. I cannot decrypt files as demonstrated below. $ sops --version sops...
GitHubmozilla

Integration Recipes

Many tools provide a plugin to directly read and write encrypted SOPS file.

Terraform

Download the following provider plugin from github

carlpett/terraform-provider-sops
A Terraform provider for reading Mozilla sops files - carlpett/terraform-provider-sops
GitHubcarlpett
$ mkdir -p ~/.terraform.d/plugins
$ curl -L -o ~/.terraform.d/plugins/terraform-provider-sops_v0.5.0_darwin_amd64.zip \
"https://github.com/carlpett/terraform-provider-sops/releases/download/v0.5.0/terraform-provider-sops_v0.5.0_darwin_amd64.zip"

$ unzip ~/.terraform.d/plugins/terraform-provider-sops_v0.5.0_darwin_amd64.zip \
-d ~/.terraform.d/plugins

$ terraform init

Then we could define a data resource that automatically decrypt SOPS json.

provider "sops" {}

data "sops_file" "secrets" {
  source_file = "secrets.enc.json"
}

## Using
provider "aws" {
  region = "us-west-2"
  access_key = data.sops_file.secrets.data["aws_access_key"]
  secret_key = data.sops_file.secrets.data["aws_secret_key"]
}

Encrypted Private Keys

SOPs works with any unstrutured files as well. The data will get encoded into a data key in a resulting json automatically.

$ sops -e private_key > private_key.sops
$ cat private_key.sops
{
  "data": "ENC[AES256_GCM,data:bVr....."
  "sops:: { ... }
}
$ rm private_key

Using the keys, we could just pipe the decrypted result to ssh-add without writing to file first.

ssh-add - <<< $(sops -d private_key.sops)

Python Script

SOPS used to be written in python, but reimplemented in golang. The pip package exists but with many features missing. It is probably better to call it via subprocess.

import subprocess
b = subprocess.check_output(['sops', "-d", "private_key.sops"])

Kubernetes Secrets

Create a new yaml file, but indicates to SOPs that only data and stringData are keys to encrypt

$ sops --encrypted-regex '^(data|stringData)$' secrets-mysecret.yaml

apiVersion: v1
kind: Secret
metadata:
    name: mysecret
type: Opaque
stringData:
  mySecret: hello123

Apply by pipe the decrypted output to K8s

sops -d secrets-mysecret.yaml | kubectl -n workflow apply -f -