## Why using etcd encryption

We recommend using etcd encryption to provide an additional layer of data security for your cluster.

Read more about it: [Kubernetes Docs: Encrypting Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)

## Enabling etcd encryption in Autopilot

To enable it in Autopilot, one possible approach is changing the `spec.topology.variables` of your cluster.yaml or Cluster resource within the management cluster and add:

```yaml
- name: etcdEncryptionSecretRef
  value:
    enabled: true
```

And create the related secret:

```yaml
/// etcd-encryption-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: etcd-encryption
type: Opaque
stringData:
  config: |-
    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: <token>
        - identity: {}
```

To generate a random value to replace `<token>`, you can use:

{% terminal height="5rem" steps="[\"head -c 32 /dev/urandom | base64\"]" /%}

Then save the above file as `etcd-encryption-secret.yaml` and run:

{% terminal height="5rem" steps="[\"kubectl apply -f etcd-encryption-secret.yaml\"]" /%}

Direct verification of encrypted Secret data in etcd (for example using `etcdctl`) requires access to the control plane / etcd.

## Encrypt already existing Secrets

Enabling encryption does not automatically rewrite Secrets that were already stored before encryption was enabled. Existing Secrets are encrypted after their next write.

To re-encrypt one existing Secret, trigger a small write (for example by annotating it):

{% terminal height="9rem" steps="[{\"command\":\"kubectl annotate secret <secret-name> \\\\\\n-n <namespace> \\\\\\nencryption-rewrite-timestamp=\\\"$(date +%s)\\\" \\\\\\n--overwrite\"}]" /%}

This updates the Secret and causes it to be written again using the currently configured encryption settings.

To re-encrypt all existing Secrets cluster-wide, run:

{% terminal height="6rem" steps="[{\"command\":\"kubectl get secrets --all-namespaces -o json | \\\\\\nkubectl replace -f -\"}]" /%}

This command reads and rewrites all Secrets with the same data, which applies encryption at rest to objects that were previously unencrypted.

## Troubleshooting

The official Kubernetes verification procedure for encryption at rest checks the raw Secret data in etcd using `etcdctl`. To execute those commands on a node, you need access to the control plane / etcd.

If you need to troubleshoot or prove that data is encrypted in etcd, follow the upstream Kubernetes verification steps:

[Kubernetes Docs: Encrypt data at rest (verification)](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#verifying-that-data-is-encrypted)