Syself Linux

The operating system on every Syself Autopilot node, Syself Linux, is immutable. It is built once and never patched. When it has to change, the node is replaced from a fresh image. So nothing on a running node drifts, and every node is provably what Syself shipped. You never install it, configure it, or SSH in to fix it. A Linux kernel feature called dm-verity enforces this: on every disk read, the kernel rechecks the OS against a hash fixed at build time. The OS is read-only and around 350 MB.

The usual way to run Kubernetes nodes is a general-purpose Linux: several gigabytes, a package manager, and hundreds of packages that all need patching. A package manager only tracks its own changes, so you cannot prove what was changed outside it. After a while every server drifts apart, and none can be proven clean. Syself Linux has nothing to patch.

The partitions and the immutable layers#

Every node has the same partition layout.

The immutable, read-only layers sit on top of that layout. The kernel verifies them with dm-verity while the node runs. The image is split into blocks. Each block has a hash, and the hashes combine into one root hash fixed at build time. On every block read, the kernel checks the block against its hash. If a block no longer matches, the kernel fails that read.

• The OS image: the kernel, the kubelet (the agent that starts and watches containers on the node), the container runtime, systemd, and the role- and hardware-specific parts a node needs. Syself builds and verifies it as one image.
• The per-node config image: a small image with this node's hostname, machine identity, SSH host keys, and network and kubelet configuration. It is built and made immutable during provisioning, one per node.

From build to a running node#

The image is made immutable at build time. The kernel enforces that from the first block it reads at boot. Nothing on the running node can change or repair a layer.

flowchart TD
    A["Build: the image made immutable, pinned by content hash"] --> B["Shipped together as one image, every layer pinned by content hash"]
    B --> C["Provisioner: writes the OS image to disk, verifies each hash, and builds the per-node config image and makes it immutable"]
    C --> D{"Kernel verifies every block against the known hash at boot"}
    D -->|"every block matches"| E["Node boots verified and joins the cluster"]
    D -->|"block mismatch"| F["No boot, or a corruption condition at runtime"]
    E --> G["syself-agent watches the running node and raises a tamper condition if integrity fails"]

The per-node config image is the one exception to the diagram. It is unique to each node, so its hash cannot be fixed at build time. The provisioner computes it on the node during provisioning.

The model is keyless. Authenticity comes from naming every layer by its content hash. The build fixes the OS image's hash. The provisioner computes the config image's hash and records all of them on the node, and the control plane (the part of the cluster that manages the nodes) knows what it handed each node. There is no signing key and no signature, so there is nothing to rotate or renew.

One image, tailored to each node#

Every node runs from the same image. Each node uses only the parts its role and hardware call for, and the rest stays inactive.

• A worker with no special hardware runs the plain operating system.
• A control-plane node adds the control-plane tooling.
• A machine with an NVIDIA GPU comes up ready for GPU work, with the driver already in place.
• A machine that supports hardware virtualization can run workloads in stronger isolation, through the secure runtime class.

Because there is one image and not a separate build per role or hardware type, nothing can drift between them.

The runtime stack#

Containers run on crun by default, with runc as a fallback. crun starts faster and uses less memory. A workload that needs more isolation than a shared kernel gives can, on bare-metal nodes with hardware virtualization, run in its own lightweight VM. The workload opts into that isolation by asking for the secure runtime class instead of the default shared-kernel runtime.

Beyond the runtime choice, every workload gets the same built-in protections, with no configuration on your side.

Every container runs behind a default system-call filter. A system call is how a program asks the kernel to do something. A container that brings no filter of its own gets the safe default, which blocks a large set of the riskiest calls.

AppArmor is on by default. It is a Linux security module that limits what each container can touch. containerd applies its default profile to every non-privileged container.

Every kernel module in the image is signed at build time, the NVIDIA modules included. The signing key is created inside the build and thrown away with it. The kernel refuses unsigned modules, so a rootkit that installs itself as a kernel module cannot load.

What happens on tampering#

When integrity fails, the node agent raises a condition. Each kind gets its own response.

VerityCorruption means the kernel caught a bad block on an immutable layer. A cloud node reboots and replaces itself once the condition holds about a minute. A bare-metal node goes to a human, because a bad read there is as likely a dying disk as an attack.

SealedOSTampered means the OS image or the config image no longer matches its recorded hash. The node keeps running so the evidence survives, and it is flagged for investigation.

NodeTampered means a protected file in the writable area changed when it should not have. It is flagged for investigation. The condition latches, so putting the file back does not clear it.

The rule for a tampered or broken node is to reprovision from the image, not repair it. An immutable layer changed out-of-band fails dm-verity on the next verified boot.

Check a node's integrity yourself#

You can ask the kernel directly whether a node still matches what Syself shipped. There is one verity device for the OS image and one for the config image, and the kernel reports, per device, whether every block is being hash-checked and whether any has failed. You can also confirm the OS image's hash agrees in all three places it appears: what the kernel enforces, what the node recorded at provision time, and what the boot line booted with.

Anyone can recompute the OS image's hash from the published image with standard tools and compare, because the build is reproducible at the image level. The reference does not depend on the node. It is not a bit-for-bit rebuild from source; the guarantee is that the image you can download is the image the node runs. Full walkthrough: Verify node integrity .

What Syself Linux does not cover#

The root of trust is software-only, on purpose. There is no Secure Boot, no measured boot, and no remote attestation. Hetzner servers do have a TPM, but Syself does not use it. A hardware-backed root of trust is out of scope here.

The anchor is the plaintext boot line on the writable EFI partition. It holds the root hash the kernel enforces. Whoever can rewrite that boot line controls what the node boots; below that line, the kernel only enforces whatever the boot line names. Syself handles this gap with process instead of cryptography: the provisioner records each node's expected hashes, the node agent checks the running kernel against them, and a drifted node is reprovisioned. If your threat model demands a hardware root of trust, this is the gap to weigh.

A couple of smaller gaps remain. The writable partition is tamper-evident, not tamper-proof, because it has to stay writable, and the baseline gap in the callout above applies to it. And one path is not locked down yet: a root process can still load an unsigned kexec image (kexec boots a new kernel without going through a full reboot).

How it compares to other node OSes#

Syself Linux is not the only minimal node OS. There are two groups.

Bottlerocket and Talos are the closest: small, immutable operating systems built only to run containers, with no package manager to patch. Syself Linux shares that goal and differs in two ways.

The first is integrity. Syself Linux is keyless: authenticity comes from the content hash of each layer, so there is no signing key to guard, rotate, or lose.

The second is the bigger one, and it is about where the OS sits. Talos is managed as a node. Each node runs its own API, and you change a node by pushing a new machine config to it; some changes apply to the running node, others reboot it. That is a coherent design: an immutable node OS for Kubernetes, with an API to drive each node.

Syself started from the other end. We did not set out to build a node OS and then manage it. We set out to have software manage a whole system, and its infrastructure, declaratively; the node OS is one part of that system. A node is never reconfigured while it runs. Its entire makeup is a fixed input, built ahead of time from your declaration, and when something has to change, Syself builds a new node and replaces the old one. There is no per-node API and nothing is assembled at runtime.

Think of two ways to change a car. Talos lets you adjust the car while it drives: some adjustments take effect at once, some need a quick stop. Syself builds the car to a spec, and to change it you get a newly built one to the new spec instead of adjusting the one on the road. Talos's way is convenient and works well. Syself's way means a running node is always exactly a built-and-tested input, with nothing changed underneath it, which is what keeps drift and runtime surprises out.

Because the OS is one part of a system Syself owns end to end, Syself also owns what usually falls to you with an OS you assemble yourself: the components in the image and how they are tuned to the hardware, the supply chain and the CVE tracking behind every one of them, the release process that tests the whole combination, the backporting of fixes onto a supported line, and the SLA to respond when something breaks. Bottlerocket and Talos hand you the OS; running the rest is yours. Syself ships it as one tested piece, versioned together in a cluster stack.

RKE2 and k3s are a different thing. They are Kubernetes distributions, not node operating systems. They package and simplify Kubernetes, but they still run on a general-purpose Linux that you own and patch. Syself Linux replaces that host.

Operating Syself Linux nodes#

Syself watches CVEs across the kernel, the runtimes, and the rest of the image, and ships a fixed build. You do not patch nodes yourself.

Treat every node as replaceable. When an integrity condition fires, decide the response, which is usually to reprovision the node. If a VerityCorruption fires on a cloud node, the platform has already replaced it. If a tamper condition fires, read the surviving evidence, then reprovision.

If you need evidence that a compromised node cannot erase, ship condition transitions and audit logs off-node before you need them. Nothing does that for you by default.

Next: node lifecycle and self-healing is what runs when a condition fires, and you can verify a running node's integrity yourself .