A node in your cluster is flagged by an integrity condition. Respond in order: preserve the evidence, decide whether to investigate, rotate what the node held, reprovision, and never boot the node writable. The order matters, because recovery destroys everything on the node.

One rule above all: **do not boot the node into any writable environment.** The server can be booted into an external rescue system, which writes to the disk that is your evidence; recovery from tampering is always a reprovision. The section [What not to do](#what-not-to-do) explains why.

## Which condition fired

The integrity conditions each tell you something different, and the platform reacts differently to each. Do not treat them as interchangeable.

| Condition          | What it means                                                                                                                                                                                                 | What the platform does                                                                                                                                                        |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `NodeTampered`     | A protected file under `/var` (the writable data area: Kubernetes state, certificates, static pod manifests, executables) was added, removed, or changed since the tamper baseline.                           | Alerts. No automatic action, so the node and its evidence survive for you.                                                                                                    |
| `SealedOSTampered` | The root hash the kernel enforces for a sealed layer (the base, a role or hardware add-on, or the per-node config) no longer matches the hash the provisioner recorded for this node.                         | Alerts. No automatic action.                                                                                                                                                  |
| `VerityCorruption` | A block in a sealed OS layer failed its dm-verity check (dm-verity is the kernel feature that verifies every disk block against a hash tree on every read). Disk corruption and tampering look the same here. | On cloud pools: reboot, then replace, after the condition holds for 60 seconds. On bare-metal pools: alerts, because a failing read is as likely a failing disk as tampering. |

This split is deliberate. Verity corruption on a cloud node is fixed by laying the image down again, and waiting only keeps a node with an unverifiable OS serving traffic, so the machine is replaced automatically. `SealedOSTampered` and `NodeTampered` are evidence of a possible intrusion, so they page a human instead of destroying the scene. The full catalog is in [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions).

Two properties of the `NodeTampered` baseline change how you respond. A clean condition on a node that rebooted after a suspected change is not proof of a clean node: the daemon re-takes its baseline from the current disk on restart, so a change made while it was down gets adopted. And root on the node can clear the finding, because the daemon writes it with the node's own kubelet credential, so treat the copy you shipped off-cluster as the record that holds. The mechanism behind both is in [Node and OS security](/docs/hetzner/apalla/security/node-and-os-security#tamper-detection-at-runtime).

> [!NOTE]
> On a cloud node, `VerityCorruption` means the machine is already being replaced. You have at most the 60-second hold plus the remediation window to capture anything from the node itself. This is why evidence shipping has to be running before the incident, not set up during it.

## Before an incident: turn on log shipping

Nothing ships logs off a node by default. Local audit logs are tamper-evident, not tamper-proof: a root process can delete them, and a reprovision erases them completely. If shipping is not running when a condition fires, the on-node evidence is one reprovision away from gone.

Set up a log shipper as a DaemonSet with write-only credentials to an append-only store, and confirm the pipeline works by testing that a `NodeTampered` transition reaches your SIEM (Security Information and Event Management system, the tool that collects and searches security events). [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) walks through the setup. Do it now, before an incident, not during one.

<Steps>
<Step>Preserve the evidence</Step>

Everything in this step reads through the Kubernetes API. Do not log in to the node yet, and do not reboot, drain, or delete anything.

Capture the condition and its transition time:

```console
$ kubectl get node <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason} {.lastTransitionTime}{"\n"}{end}' \
  > node-conditions.txt
```

Capture the tamper report. For `NodeTampered`, the `tamper` section lists the paths that changed and how (added, removed, or modified):

```console
$ kubectl get node <name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/tamper}' \
  | jq . > node-tamper-report.json
```

Capture the node's events and the pods that were scheduled there:

```console
$ kubectl get events --all-namespaces \
  --field-selector involvedObject.name=<name> -o json > node-events.json
$ kubectl get pods --all-namespaces \
  --field-selector spec.nodeName=<name> -o json > node-pods.json
```

Store all of it off-cluster, in the same append-only store your logs ship to. If shipping was already running, the node's auditd stream and API audit log are there too; note the incident window so you can pull them later.

Then cordon the node so no new workload lands on it. Cordon marks the node unschedulable without touching running pods, so it does not destroy anything:

```console
$ kubectl cordon <name>
```

<Step>Decide, replace or investigate</Step>

For `VerityCorruption` on a cloud pool the platform has already decided: the machine is replaced. Your job is Step 1 fast, then Steps 3 and 4 for the replacement that appears.

For `SealedOSTampered` and `NodeTampered`, a human decides, and the first question is whether the change is expected. Read the `tamper` section of the report you captured. A path changed by a config push you made, or an operator you installed, is a finding to tune, not an intrusion. If every changed path traces to a change you made, you can go straight to reprovisioning (Step 4) and skip credential rotation.

If any change is not yours, treat the node as compromised and keep it isolated. You can confirm the sealed-layer state from outside the node, and even from on the node over SSH, because reading does not alter the disk: [Verify node integrity](/docs/hetzner/apalla/security/verify-node-integrity) shows how to compare each layer's enforced root hash against the recorded values. What you cannot do is repair anything you find in place. A sealed layer edited from a rescue system fails verification on the next sealed boot, so investigation ends in a reprovision either way. The decision is only how much evidence to collect first.

If your incident process needs disk-level forensics, capture a raw image of the disk without mounting any filesystem. Mounting is a write: even a default "read-only" mount can replay the filesystem journal onto the disk. Image first, examine the copy.

<Step>Rotate what the node held</Step>

Assume everything readable on the node was read. Rotate before you reprovision, because reprovisioning removes your ability to check what was there.

- **Secrets mounted into pods on the node.** Every Secret mounted by the pods you captured in Step 1 (`node-pods.json`) was present on the node in plaintext. Rotate each one upstream: create the new credential in its source system, update the Kubernetes Secret, restart the workloads that read it, then revoke the old value. [Manage application secrets](/docs/hetzner/apalla/security/manage-application-secrets) covers the sequence.
- **ServiceAccount tokens of those pods.** Restart the affected workloads so they receive fresh bound tokens, and review your audit logs for API calls made with the old identities during the incident window.
- **Anything your workloads wrote to disk.** Local volumes, emptyDirs, and application state on the node were readable too. Rotate any credential your applications persist.
- **On a control-plane node: every Secret in the cluster.** Control-plane nodes hold etcd, and etcd holds all Secrets. If etcd encryption at rest is not enabled, a compromised control-plane node means cluster-wide Secret rotation. Encryption at rest protects the on-disk copy, but a process running as root on a live control-plane node is beyond what it can protect against.

The node's own platform identities need no action from you: kubelet and tunnel certificates are short-lived and rotate automatically, and the replacement node bootstraps fresh ones.

<Step>Reprovision</Step>

Recovery is a reprovision, never a repair in place. This is how the platform treats any node that drifts, tamper or not: see [the node lifecycle](/docs/hetzner/apalla/concepts/operations/node-lifecycle). A fresh node is laid down from the sealed image, boots through the same dm-verity verification as every other node, and starts with a clean tamper baseline. Nothing from the old node survives onto it.

Drain the node so workloads move off in an orderly way. Drain evicts pods with a termination grace period and respects PodDisruptionBudgets:

```console
$ kubectl drain <name> --ignore-daemonsets --delete-emptydir-data
```

Then contact Syself to replace the machine. On cloud pools the replacement is a new VM. On bare-metal pools the same physical server is wiped and reinstalled; local bare-metal volumes on other disks survive, so if the incident makes you distrust that data, delete those volumes as part of the response.

When the replacement joins, confirm it is clean: `SealedOSTampered`, `NodeTampered`, and `VerityCorruption` all `False`, and the sealed layers verified as described in [Verify node integrity](/docs/hetzner/apalla/security/verify-node-integrity).

</Steps>

## What not to do

**Do not boot the node into a writable environment.** Booting the server into an external rescue system gives a writable disk, and it is a mistake here for two reasons. It writes to the disk you want to keep as evidence, and it cannot fix anything anyway, because a sealed layer edited in place fails verification on the next sealed boot.

**Do not "clean up" and keep the node.** Removing an attacker's file does not clear the latched condition, does not remove whatever you did not find, and leaves you running a node you have already decided not to trust. Sealed nodes are replaced, not repaired.

**Do not reprovision before Steps 1 and 3.** Reprovisioning wipes the OS disk. Local logs, the attacker's artifacts, and your ability to enumerate what was exposed all go with it. The order here exists because the destructive step is last.