Fast, Reliable Provisioning

Most tools build a Kubernetes node while it runs, configuring a live server step by step. Syself Autopilot builds the whole node before it boots. The OS, the config, the certificates: all of it is written while the node is offline. Then Syself checks the finished disk against a known hash and restarts the machine. The node comes up fast, and a broken build never joins the cluster.

The problem with configure-on-boot#

Cloud-init and Ansible build a node at boot. They run a set of scripts after the OS comes up: fetch secrets, install packages, join the cluster. These scripts run on a machine that is already live, and any step can fail on a node the cluster already treats as a real node.

These failures are common. A package mirror times out. A script exits halfway. The node holds an IP but never joins the cluster. Nothing crashes, so no alert fires. You find the dead node later, still being billed for it.

Syself moves all the building off the live node, so this failure window does not exist. The work runs in the Hetzner rescue system, a small Debian that runs entirely in RAM. There it writes the OS to disk, stages the full Kubernetes config, and checks every byte. When the machine reboots, the rescue system disappears, leaving only the finished disk.

One image, split by role and hardware#

Everything comes from a single image, one version per stack release. There is no separate image per role and no separate GPU image. The rescue system decides which pieces a node gets, based on what Syself's provisioning finds on the machine:

• The base image, always. This is the shared operating system, one read-only image fixed at build time. It is protected by dm-verity, a Linux kernel feature that checks every disk block against a known hash on each boot. A worker node is this base and nothing more.
• The control-plane tooling, when the config marks the node a control plane. It carries the extra tools a control plane needs, including its health check and a local copy of the control-plane container images.
• The NVIDIA driver, when the machine has a GPU, so it comes up ready for GPU work.
• The secure runtime class (Kata), when the hardware supports virtualization. It runs workloads in their own lightweight virtual machines for stronger isolation.

The control-plane container images are built into the image, so they sit on local disk at first boot. A fresh control plane pulls nothing from the network as it comes up. A registry outage or blocked egress cannot stall it.

How Syself builds each node#

Syself's provisioning runs from the management cluster and drives each machine through its build. It boots the machine into the rescue system and starts the work there. The build is one straight sequence. If a step fails, the build stops before the next step starts. A machine that fails a check is never left half-built.

flowchart TB
  a["Boot the machine into the Hetzner rescue system"] --> p1["Check the machine: detect role and hardware, gate on disk health and speed"]
  p1 --> p2["Write the verified OS image to disk, check every block against a known hash"]
  p2 --> p3["Stage the per-node Kubernetes config, certificates, and cluster CA"]
  p3 --> p4["Fix the small per-node layer and record the boot-time verification hash"]
  p4 --> p5["Unmount, confirm the disk, reboot"]
  p1 -. "failed gate" .-> x["Build aborts: the machine never becomes a node"]
  p2 -. "hash mismatch after retries" .-> x

1. Check the machine. Work out which pieces this machine needs, by role and hardware. Then run the checks before touching the disk: the right tools are present, the firmware is correct, the target disk is large enough, the disk reports itself healthy, and a quick read test proves it is fast enough. A slow disk gets rejected here, not weeks later as an etcd member that cannot keep up.
2. Write the OS. Stop if another disk holds an unknown operating system, so the wrong disk never gets wiped. Lay out the partitions, write the immutable OS image, and check every write against the hash the build recorded. A mismatch gets retried. Data that fails the check never ends up on a booted node.
3. Stage the per-node files. Put each node's own files in place: the Kubernetes config, the DNS and network settings, the SSH host keys, the hostname, and the labels. The slow Kubernetes setup steps run here whenever they can, so first boot skips them. A node joining an existing cluster gets that cluster's trusted certificate staged ahead of time, so it never has to exchange a join token at runtime.
4. Fix the per-node layer. The OS image is already fixed at build time. The only part fixed on the node itself is the small per-node layer: hostname, machine identity, SSH keys. It is quick, and the kernel verifies it on every boot, just like the base.
5. Finish and reboot. Unmount, confirm the disk is clean, and reboot into the finished OS.

Every step reports its state back to the platform, so you can see when a machine stops. On bare metal, the machine moves through named stages you can read with kubectl, from preparing through to provisioned. A failure records which step stopped, for example the rescue system being unavailable or a disk that could not be wiped. Cloud machines report their own coarser stages. You can read the machine's exact stage instead of searching for a node that never came up.

Speed#

There is no build left to do at boot: no decompression, no file-by-file copy, no package install. The image is already written and verified. The control-plane images already sit on disk. The slow Kubernetes setup steps already ran in the rescue system. At boot, the node only joins the cluster from the config staged for it.

In practice this runs about 60 percent quicker than building a node at boot. A single node is ready in about three minutes, and a full highly available cluster in about nine. These are approximate figures, not guarantees.

Reliability#

Each gate removes a whole class of failure before it can reach a running cluster: a degraded disk, the wrong disk, a corrupt write, a half-built machine. None of them turn into a node.

Every node is built from the same image in the same known environment, so there is no per-node variation to explain later. Each build runs on its own machine in its own rescue system, so a whole pool builds in parallel: ten nodes come up in about the time one takes.

Security#

Only a pre-built image lands on the disk. Every layer is checked against the hash Syself recorded at build time, so only the exact image Syself shipped ever boots. The running OS is read-only, so nothing mutable gets installed at runtime and there is no configure-on-boot step for the network to tamper with.

From first boot to Ready#

At first boot the node reads the config staged in the rescue system and joins the cluster, skipping every step that already ran before boot. After that, the node comes up in a fixed order, and each step has to finish before the next can start.

sequenceDiagram
  participant C as "Provisioning controller (management cluster)"
  participant R as "Rescue system (throwaway, in RAM)"
  participant N as "Node: first boot + kubelet"
  participant CI as "Cilium"
  participant M as "Hetzner CCM"
  C->>R: "Boot rescue, pass the image reference and bootstrap config"
  R->>R: "Write the image, verify each layer, fix the per-node data"
  R-->>C: "Image written and verified"
  C->>N: "Reboot into the immutable OS"
  N->>N: "First boot: join the cluster from the staged config"
  Note over N: "kubelet restarts until its config is written (expected)"
  N->>CI: "Cilium DaemonSet schedules: pod networking + host firewall"
  M->>N: "CCM sets addresses + provider ID, removes the not-initialized mark"
  N->>N: "Node Ready"
  N->>C: "kubelet requests serving cert (CSR)"
  C->>N: "Controller approves the CSR: cert signed by the cluster CA"

The kubelet starts in a mode that waits for the cloud provider to finish setting up the node. Until that happens, Kubernetes marks the fresh node as not yet initialized and keeps normal workloads off it. Cilium, the network layer, gives the node its pod networking and host firewall. The Hetzner cloud-controller-manager then fills in the node's addresses and provider ID and clears the mark. Only then does the node go Ready.

Cilium is hard to get right. Its pod networking and host firewall are tightly coupled to how Kubernetes routes traffic, the kernel modules the OS loads, and what the base image ships. Syself integrates and tests those layers together and builds the result into the image. A booted node has working networking, so you never configure Cilium yourself.

The last hand-off gives the kubelet a certificate the cluster trusts. Once it registers, the kubelet asks for its serving certificate with a certificate signing request (a CSR). Kubernetes does not approve these serving requests on its own, so a controller in the management cluster approves it. The kubelet ends up with a certificate signed by the cluster's own authority. That lets the metrics component read from the kubelet over verified, encrypted connections instead of an insecure fallback.

Limits#

• Alert on readiness, not kubelet restarts. The first-boot kubelet restarts are expected (see the note above). Watch Node readiness and Syself's health conditions instead of raw kubelet restarts.
• An extra control plane waits on the CCM. A joining control plane waits up to 10 minutes for the cloud-controller-manager to assign its node address before it can join etcd. The CCM is a cluster component that comes up after the first control plane, so this wait applies to every control plane except the first one in a new cluster.
• Provisioning needs the Hetzner rescue system. The whole build runs inside it. If Hetzner's rescue system is down, new provisioning pauses. Running nodes keep working.

One build for every job#

The same gated build does every job: it stands up a fresh cluster, repairs a broken node, and upgrades a node, running the same way every time, so a throwaway or preview cluster is as cheap to stand up as a production one. It runs automatically for self-healing whenever a node fails. What lands on the disk is one cluster stack ; how the build adapts to dedicated servers versus cloud VMs is in bare metal, cloud, and accelerators , and running control planes on dedicated servers is in fully bare-metal clusters .