etcd Encryption

Why encrypt etcd

etcd stores every Kubernetes Secret at rest. Encrypting it means a stolen disk or backup does not hand over your Secrets in plaintext.

Read more about it: Kubernetes Docs: Encrypting Data at Rest⁠

Enabling etcd encryption in Syself Autopilot

To enable it in Syself Autopilot, one possible approach is changing the spec.topology.variables of your cluster.yaml or Cluster resource within the management cluster and add:

		- name: etcdEncryptionSecretRef
  value:
    enabled: true
	

And create the related secret:

		apiVersion: v1
kind: Secret
metadata:
  name: etcd-encryption
type: Opaque
stringData:
  config: |-
    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: <token>
        - identity: {}
	

To generate a random value to replace <token>, you can use:

		$ head -c 32 /dev/urandom | base64
	

Then save the above file as etcd-encryption-secret.yaml and run:

		$ kubectl apply -f etcd-encryption-secret.yaml
	

Direct verification of encrypted Secret data in etcd (for example using etcdctl) requires access to the control plane / etcd.

Encrypt already existing Secrets

Enabling encryption does not automatically rewrite Secrets that were already stored before encryption was enabled. Existing Secrets are encrypted after their next write.

To re-encrypt one existing Secret, trigger a small write (for example by annotating it):

		$ kubectl annotate secret <secret-name> \
-n <namespace> \
encryption-rewrite-timestamp="$(date +%s)" \
--overwrite
	

This updates the Secret and causes it to be written again using the currently configured encryption settings.

To re-encrypt all existing Secrets cluster-wide, run:

		$ kubectl get secrets --all-namespaces -o json | \
kubectl replace -f -
	

This command reads and rewrites all Secrets with the same data, which applies encryption at rest to objects that were previously unencrypted.

Troubleshooting

The official Kubernetes verification procedure for encryption at rest checks the raw Secret data in etcd using etcdctl. To execute those commands on a node, you need access to the control plane / etcd.

If you need to troubleshoot or prove that data is encrypted in etcd, follow the upstream Kubernetes verification steps:

Kubernetes Docs: Encrypt data at rest (verification)⁠