What you can and cannot change on a sealed node
The OS on every node is read-only and sealed with dm-verity , a kernel feature that checks every disk block against a stored hash before use. On a running node you cannot install a package, edit /usr or /etc, or load a kernel module. Know that boundary before you try to change anything.
The layers#
A node is built in layers, from most sealed to most writable.
| Layer | What it holds | Mutability |
|---|---|---|
| Base image | The OS: /usr, /bin, libraries, the kernel | Read-only, dm-verity sealed at build time. Identical on every node. |
| Optional add-on components | Components for control-plane and GPU nodes that merge into /usr and /opt | Read-only, sealed, and verified, pinned by content hash |
| Per-node config | /etc: hostname, machine-id, SSH host keys, network and kubelet config, authorized_keys | Built per node during provisioning, then sealed and verified like the rest of the OS |
Writable /var | Your container data, logs, local volumes, cluster PKI | Read-write, tamper-evident but not sealed |
/, /usr, /opt, /bin, and /etc are read-only. /var is the writable, persistent partition. /run and /tmp are tmpfs, cleared on every boot.
Note
"Tamper-evident" means a change to /var is detected, not blocked. The node watches security-relevant files and any new executable under /var, and a change raises the NodeTampered condition. The condition stays raised even after you revert the file.
What root can and cannot do#
Root on a sealed node is not root on an ordinary Linux server. The verity check rejects any OS change, even one made by root. Root can read files and run diagnostics, but cannot write to the OS.
| You can | You cannot |
|---|---|
Read all files, inspect logs, read /var | Edit /usr, /etc, or any OS file |
Run diagnostics: journalctl, ip, ss, crictl, dmsetup | Install packages (there is no package manager) |
Run a tool you copy into writable /var | Add a permanent systemd unit (sealed /etc) |
Collect a bundle with syself-log-collector | Change the sealed OS in place |
Run your own software#
Run it in a container. A normal pod covers almost every case: the kubelet and containerd manage it, and nothing on the host changes. For host-level access, add hostNetwork, hostPID, a hostPath volume, or securityContext.privileged: true to the pod. This is the supported way to run your own code on a node, and it changes nothing about the OS.
The OS itself is managed for you. You do not add software to it, and you do not build or roll out node images. That is fully automatic.
Note
For a one-off diagnostic you can copy a binary into writable /var and run it for this boot only. The node notices: the file sits outside the dm-verity seal, is recorded in the node's own annotation, and raises the NodeTampered condition. A node changed this way may not be fully supportable, depending on what changed, so keep it to short-lived debugging and reprovision the node afterward.
Tip
Do you have something that genuinely needs to live on the node? Tell us. If it is a capability that would help other users too, email Syself with your use case and we can look at making it a native platform feature. If it is specific to your setup, reach out anyway, and we will help you find a supported way to solve it so we can still fully support your cluster.
One boot entry for a healthy node#
A healthy node boots from a single dm-verity-verified entry, with root mounted read-only. There is no verity-off boot option. Nothing you change on the running system survives a reboot.
Important
If a node cannot boot its sealed image, do not look for a maintenance or recovery boot entry. There is none. The fix is to reprovision the node, which reinstalls a fresh, verified image.
What happens when a check fails depends on which one:
| Condition | Trigger | What happens |
|---|---|---|
VerityCorruption | A dm-verity block read on the base fails | A cloud node reboots, then is replaced if the condition persists |
SealedOSTampered | A sealed OS layer's hash no longer matches the value recorded at provisioning | Raises an alert for a human to review |
NodeTampered | A file changes under /var | Raises an alert for a human. It stays raised until the node is reprovisioned. |
Check the dm-verity state directly with dmsetup:
$ dmsetup status root # a trailing 'V' means the layer is verified
A NodeTampered finding is also written to the autopilot.syself.com/tamper annotation on the node's own Node object, so it survives even if the node's disk is wiped afterward. It names every changed path with its kind (added, removed, changed) and category:
$ kubectl get node <node-name> -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/tamper}'
SealedOSTampered has no annotation of its own. Read its condition message instead:
$ kubectl get node <node-name> -o jsonpath='{range .status.conditions[?(@.type=="SealedOSTampered")]}{.status}{" "}{.message}{"\n"}{end}'
The health daemon publishes one annotation per domain, refreshed every two minutes. See Node labels and annotations for the full set.
Where change actually happens#
Because you cannot mutate a node, real change happens one layer up:
- What the OS contains is fixed by the node image Syself publishes. It changes only when Syself ships a new OS version, which Syself Autopilot rolls out by replacing nodes. You do not build or edit the image.
- To change how a node is configured, change the cluster variable that drives it (machine type, region, placement group, labels) and re-apply. Syself Autopilot reconciles it.
This is the "replace, not patch" model: change happens by rolling a new node, never by patching a running one.
Why the seal exists#
An immutable, verified OS cannot drift. Every node of a given image is byte-for-byte identical and provably unmodified. That is what makes reprovisioning safe and upgrades predictable.
The seal is keyless: there is no signing key, only a root hash. Anyone can recompute that root hash from the published image, using stock tools. Compare it to the hash the node reports. A match confirms the node runs exactly the image Syself published. For the full design, see Syself Linux .
If a node stops behaving the way you expect, start by debugging it before reaching for emergency SSH access .