Verify node integrity

To trust that every node in your cluster runs exactly the OS that Syself built and nothing else, check its tamper conditions from the Kubernetes API, then confirm its live dm-verity root hash against the published value by hand.

What makes a node trustworthy#

Syself Linux is read-only. The operating system cannot be changed while a node runs. There is no package manager and no writable place in the OS to install anything. A process or an attacker cannot make a permanent change to the running OS.

The OS is also integrity-checked at runtime. Every block is verified against a hash that was fixed when Syself built the image. This uses dm-verity, a Linux kernel feature that checks each disk block before the kernel reads it. If any block does not match, the check fails. Every sealed layer works this way: the base OS, each add-on layer a node's role or hardware calls for, and the per-node config layer, which holds /etc.

The build is reproducible at the image level, so anyone can recompute the OS image's hash from the released image with stock tools and compare. This is not a bit-for-bit rebuild from source; the guarantee is that the image you can download is the image the node runs.

A node whose OS was changed in place fails the kernel's block check the moment the changed block is read, and the health daemon raises VerityCorruption. A node whose layer was re-sealed under a new root hash does boot and run; what gives it away is that the root hash it enforces no longer matches the one recorded for it. So "up and serving" is not by itself proof: only the root-hash comparison is.

Your workload data is kept separate from the OS. Logs, container state, and Kubernetes data live in a writable area outside the sealed OS. Changes there are normal and expected; they do not touch the OS.

Nodes that drift are replaced, not repaired#

Nodes are disposable. The platform never repairs a node in place. If a node drifts from its expected state, the platform reprovisions or replaces it. A fresh node starts clean from the same built image.

This is why you do not need to trust a long-lived, hand-maintained machine. You only need to trust the build and the checks on the running node. A node that fails a dm-verity block check on a cloud pool is replaced automatically. Tamper findings (NodeTampered, SealedOSTampered) and verity corruption on bare metal raise an alert for a human instead, so the node and its evidence survive for investigation; recovery is still a reprovision, never a repair in place.

How the platform proves it, and how you can check#

The platform provisions each node and confirms it booted the correct image before the node joins the cluster. From then on, a node's health daemon watches for tamper signals and reports them to the Kubernetes API, so you can check the state of every node without logging in.

Check whether any node is flagged as tampered:

		$ kubectl get nodes -o json | jq -r '
    .items[]
    | select(.status.conditions[]?.type=="NodeTampered"
        and (.status.conditions[] | select(.type=="NodeTampered").status=="True"))
    | .metadata.name'
	

Check one node's tamper condition:

		$ kubectl get node <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="NodeTampered")]}{.status}{" "}{.reason}{"\n"}{end}'
	

status: True means the health daemon saw a change in a watched file since it took its baseline, which it does on its first check after node bring-up finishes. The condition stays set for the life of that daemon even if the file is put back. A daemon restart or a node reboot re-takes the baseline from the current disk and clears the condition, so False on a node that rebooted after a suspected change is not proof of a clean node. Only a fresh node gives you a baseline you can trust.

Read the detail of what changed:

		$ kubectl get node <name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/tamper}' | jq '.tamper'
	

The report lists the paths that changed and how each changed (added, removed, or modified). Check whether each change is an expected write from normal operation. If it is not, treat the node as compromised.

The platform also reports a separate signal if the OS integrity check itself ever fails on a running node. See Node health conditions for the full list of conditions.

Check the seal yourself, on the node#

You do not have to take the platform's word for it. You can log in to any node and confirm that every sealed layer is a healthy dm-verity device running the exact root hash the node was built and provisioned with.

What makes this check meaningful is not a key. The OS is keyless: there is no signing key, no signature, and no key file on the node to trust or forge. Each sealed layer is pinned by its dm-verity root hash instead. A changed layer has a different root hash. The reference you compare against is not the node itself: it is the root hash Syself publishes for the release (the build is reproducible , so you can recompute it from the released image with stock veritysetup) and the per-node value the control plane holds off the node. So for each layer you confirm two things: the kernel reports the device VERIFIED, and the root hash it is enforcing matches the value the provisioner recorded.

SSH to the node on port 100 (see Restrict SSH access ), then run the checks below.

		$ dmsetup ls --target verity
# ...                 <- one line per add-on layer and one for the per-node config layer
# root      (252, 0)  <- the base OS
	

		for dev in $(dmsetup ls --target verity | awk '{print $1}'); do
  printf '%s\t' "$dev"; dmsetup status "$dev"
done
# ...       0 ...     verity V
# root      0 2854712 verity V   <- "V" = verified; every block is being hash-checked
	

		# the root hashes recorded for this node, per layer
$ cat /var/lib/syself/sealed-layers.json
 
# the hash the kernel is enforcing for a device (base shown; repeat per device)
$ dmsetup table root | grep -oE 'sha256 [a-f0-9]{64}'
# sha256 d70a2dec9102895a5afd3193dc79786af8ad770188abd7bee9ef9bc4295209d5
	

		$ grep -oE 'sha256 [a-f0-9]{64}' /proc/cmdline
# must equal the base entry in sealed-layers.json and the dmsetup table root hash
	

If every device shows V and every enforced root hash matches its recorded entry, this node is running the exact base, add-on layers, and config that were built and provisioned for it. If any layer had been altered, its root hash would be a different value and would not match the recorded entry; for a running layer the kernel would also refuse the changed block, so the device would show C or the node would not have booted. You can check for yourself that the match matters: pick any layer, change one character of the recorded hash in your own copy, and the comparison fails.

Because the build is reproducible , you can also recompute the base root hash from the released image with stock veritysetup and confirm it matches what Syself published. That recomputation is the check that does not depend on the node: the provisioner's record and the health daemon's comparison both live on the node itself.

Why a change to a sealed layer cannot be hidden#

The important guarantee is not that a node cannot be changed. You own the hardware, so nobody can stop you from rewriting a disk. The guarantee is that a change to a sealed layer cannot be hidden, and here is what makes that hold:

1. The sealed layers are read-only at runtime. dm-verity fails any modified block, so you cannot alter a running node's base, add-on layers, or per-node config in place.
2. Changing a sealed layer means re-sealing it, which changes its root hash. Someone with full access can boot the rescue system and rewrite a layer. Full access means your SSH key, Hetzner's rescue system, or physical access to the disk. But dm-verity then demands a hash tree that matches, so they must compute a new root hash and record it for the layer to open. The tampered layer now opens, but under a different root hash than the one Syself published and the control plane recorded for this node.
3. The reference lives off the node. The node holds no key, so an attacker has no signature to fake there. What a layer should hash to is fixed by its content: the published release hash, which anyone can recompute from the released image with stock veritysetup, and the per-node value the control plane holds. A tampered layer has a new hash that matches neither.

So a re-sealed layer passes its own dm-verity (they sealed it themselves), but its root hash is not the one the provisioner recorded for this node. Checked against that record, or against the published release hash, the mismatch shows.

The one thing a disk-owning attacker controls that dm-verity does not stop is the boot line itself. The kernel builds the base's verity device from the dm-mod.create= line on the writable EFI partition. Whoever can rewrite that line chooses what the node boots. That is the trust boundary: everything below the boot line is enforced by the kernel, and the boot line itself is the anchor. There is no Secure Boot or TPM to pin it in hardware, and that is out of scope, not planned. What closes the gap in practice is operational, not cryptographic: Syself owns provisioning end to end, the provisioner records each node's expected root hashes and the health daemon checks the running kernel against them, the released image lets anyone recompute the base hash independently, and a node that drifts is reprovisioned or deleted, never repaired in place.

This is a supply-chain property, end to end: the build fixes each layer's root hash, the registry carries the digest-pinned artifact, the provisioner records the hash onto the node, and anyone can verify the running node against the published hash. No party in that chain (the registry, the provisioner, you as the customer with full root and full access to your own Hetzner account, or the infrastructure provider) can substitute a modified layer without changing a root hash that both the provisioner's record and a recomputation from the released image pin.

What to do when a node is flagged#

Treat the node as untrusted, then work the runbook. Respond to a tampered node walks the full sequence: preserve evidence, decide whether to investigate, rotate what the node held, and reprovision. Sealed nodes are built to be replaced, not repaired, so a fresh node starts clean.

One thing to get right before you touch the node: send the NodeTampered event and the report to your SIEM (Security Information and Event Management system, a tool that collects and stores security events). That keeps the evidence before the node is gone. See Retrieve audit logs for how to ship evidence off-node.

Summary#