A Syself Autopilot node is never patched. When it has to change, it is replaced by a fresh, identical node built from a tested image.

Most platforms work the other way. A server is patched in place: it runs an update tool, applies fixes on its own schedule, and reboots. A year later, no two servers are quite the same. Each one got a different patch history, or a manual edit during some outage. Those small differences cause the next failure: a fix that works on one server fails on another.

A Syself Autopilot node cannot drift like this, because it never changes at all. Its operating system is immutable: it cannot change after the image is built. There is nothing to patch on a running node, and you can check that a node runs exactly what Syself shipped.

|                                 | Patch in place                                      | Replace from an immutable image (Syself)                      |
| ------------------------------- | --------------------------------------------------- | ------------------------------------------------------------- |
| How a fix arrives               | A package update and a reboot, run on every node    | A new immutable image; nodes are replaced one at a time       |
| Node state after a year         | Base OS plus that node's own patch and edit history | Byte-identical to the image it booted from                    |
| "What is running on this node?" | Log in and inspect, then trust the answer           | Answered by the image hash the kernel enforces on every read  |
| A change that fails halfway     | A node in an unknown mixed state                    | The bad node never joins; running nodes are untouched         |
| Difference between two nodes    | Grows over time                                     | None; every node comes from the same hash-pinned image        |
| An edit made during an incident | Goes unnoticed and the fleet drifts apart           | Rejected by the read-only OS, or flagged, and gone on replace |

## Immutable, reproducible nodes

Every node runs one build image, control plane and worker alike. It is packaged and downloaded like a container image, one version per release. It contains the whole operating system as one image: the kernel, kubelet, the container runtime, and systemd, plus only the extra pieces a node's role or hardware needs, like the control-plane tools, a GPU driver, or support for extra-isolated containers. A plain worker gets just the operating system.

The image version is a hash of the exact parts pinned inside it (kernel, container runtime, kubelet, etcd, and the CNI plugin that gives pods their network). The same parts always produce the same image, so two nodes on the same version are byte-for-byte identical. There is no signing key; the hash is the version. The base stays read-only for the life of the node, and the kernel checks each read against a recorded hash. The full mechanism, what happens on tampering, and how you check a node yourself are in [Syself Linux](/docs/hetzner/apalla/concepts/internals/syself-linux).

## How a node is replaced

A node is never patched. To change it, you cordon and drain it, then replace it with a fresh node written from the image. That image is written to disk before the node ever boots, so a starting node installs nothing and configures nothing; how that works is [fast, reliable provisioning](/docs/hetzner/apalla/concepts/operations/fast-reliable-provisioning).

```mermaid
stateDiagram-v2
    ImmutableImage: Immutable image (hash-pinned)
    Provisioning: Provisioning (Hetzner rescue system)
    RunningImmutable: Running immutable (read-only, hash-verified)
    Cordoned: Cordoned (unschedulable)
    Drained: Drained (pods evicted)
    Replaced: Replaced (machine deleted)

    [*] --> ImmutableImage: Syself builds and fixes a release
    ImmutableImage --> Provisioning: server boots into rescue, image written and verified
    Provisioning --> RunningImmutable: reboot into the immutable OS
    RunningImmutable --> Cordoned: upgrade, scale-down, or unrecoverable failure
    Cordoned --> Drained: evict pods, respecting PodDisruptionBudgets
    Drained --> Replaced: delete the node
    Replaced --> Provisioning: fresh server, same or newer image

    note right of RunningImmutable
        Never patched in place.
        No package manager,
        no config changes,
        no edit-and-rebuild path.
    end note
```

The sequence is ordinary Kubernetes. First the node is cordoned: Kubernetes stops placing new pods on it. Then it is drained: the pods already running move elsewhere, each asked to shut down within a short grace period. The drain respects PodDisruptionBudgets. That is a rule for how many copies of an application must stay running. The drain waits rather than break the rule, but only for a few minutes. Then the machine is deleted and a fresh node takes its place.

The same loop handles every kind of change. A critical CVE fix (the fix for a publicly disclosed security vulnerability) arrives as a new image and runs the loop, not an emergency patch across a fleet of slightly different machines. An upgrade runs the same loop with a newer image. A failing disk runs it with the same image. Nothing builds up between replacements: no leftover config, no half-applied fix, no node that is different because of something done to it earlier.

Syself builds the image. Your job is to make your workloads safe to replace.

## Preparing your workloads for replacement

Assume any node can vanish at any moment, because from the platform's side that is the normal way to make a change.

Never store data on the OS disk. It gets rewritten on every replacement. Data belongs in a PersistentVolume. On cloud servers that is a Hetzner volume that is separate from the machine and reattaches on its own. On bare metal, local volumes are on the server's other disks. They survive a reprovision of that same server. But they do not follow a workload to a different physical server, so replicate anything you cannot afford to lose.

Never edit a node over SSH. The read-only paths reject the edit. An edit to the writable area trips a tamper baseline. Either way, the change is gone the next time the node is replaced.

> [!WARNING]
> Never SSH into a node to fix or configure it. Put permanent changes in your manifests or the `Cluster` object, where Syself reconciles them onto every node, including the ones that do not exist yet.

The rest is standard Kubernetes hygiene that matters more here. Run more than one copy of each application. Set a PodDisruptionBudget. Shut down cleanly when Kubernetes asks a pod to stop. Do those and node replacement no longer disrupts your workloads.

If a change really does belong on the node itself, Syself builds it into the image and boots a replacement node that already has it.

Replacement is normal. It is how changes happen, not a sign that something went wrong. For more, see [self-healing](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement), [cluster upgrades](/docs/hetzner/apalla/concepts/operations/cluster-upgrades), [Syself Linux](/docs/hetzner/apalla/concepts/internals/syself-linux), and [local storage on bare metal](/docs/hetzner/apalla/storage/local/local-nvme-with-topolvm).