etcd Encryption

Why encrypt etcd#

etcd stores every Kubernetes Secret at rest. Encrypting it means a stolen disk or backup does not hand over your Secrets in plaintext.

Read more about it: Kubernetes Docs: Encrypting Data at Rest⁠

Enabling etcd encryption in Syself Autopilot#

To enable it in Syself Autopilot, one possible approach is changing the spec.topology.variables of your cluster.yaml or Cluster resource within the management cluster and add:

		- name: etcdEncryptionSecretRef
  value:
    enabled: true
	

And create the related secret:

		apiVersion: v1
kind: Secret
metadata:
  name: etcd-encryption
type: Opaque
stringData:
  config: |-
    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: <token>
        - identity: {}
	

To generate a random value to replace <token>, you can use:

		$ head -c 32 /dev/urandom | base64
	

Then save the above file as etcd-encryption-secret.yaml and run:

		$ kubectl apply -f etcd-encryption-secret.yaml
	

Direct verification of encrypted Secret data in etcd (for example using etcdctl) requires access to the control plane / etcd.

Encrypt already existing Secrets#

Enabling encryption does not automatically rewrite Secrets that were already stored before encryption was enabled. Existing Secrets are encrypted after their next write.

To re-encrypt one existing Secret, trigger a small write (for example by annotating it):

		$ kubectl annotate secret <secret-name> \
-n <namespace> \
encryption-rewrite-timestamp="$(date +%s)" \
--overwrite
	

This updates the Secret and causes it to be written again using the currently configured encryption settings.

To re-encrypt all existing Secrets cluster-wide, run:

		$ kubectl get secrets --all-namespaces -o json | \
kubectl replace -f -
	

This command reads and rewrites all Secrets with the same data, which applies encryption at rest to objects that were previously unencrypted.

Troubleshooting#

The official Kubernetes verification procedure for encryption at rest checks the raw Secret data in etcd using etcdctl. To execute those commands on a node, you need access to the control plane / etcd.

If you need to troubleshoot or prove that data is encrypted in etcd, follow the upstream Kubernetes verification steps:

Kubernetes Docs: Encrypt data at rest (verification)⁠