Fast Kubernetes Node Provisioning Before First Boot

Node provisioning is a lot faster on Syself Autopilot 1.36, thanks to the new Syself provisioner. The provisioner builds each node from the Hetzner rescue system, before the node's own OS boots. It writes the sealed image to disk, stages the node's config, and reports the real state of every step back to the platform. A node is ready roughly 60% faster than before: about three minutes for a single node, or nine for a full high-availability cluster.

Provisioning is the step that turns a raw Hetzner server into a Kubernetes node. It doesn't need to build anything: the immutable image was assembled and tested back at release time, so the provisioning step responsibility is to write that finished image to disk and to stage the config the node boots with.

The old way wasn't ideal

Provisioning used to run on cloud-init in the virtual machines, or an install script on bare metal servers. Both are fire-and-forget: they run on a booting node the platform can't see into. You can test to the 99th percentile and still hit failures that only appear at scale. When one hits, you get a node that never joins and little to explain why. Nothing crashes, so no alert fires, and you find the dead server later, still being billed for it.

We moved all of that off the live node.

How does the new node provisioning work?

It runs in the Hetzner rescue system, a small environment that lives entirely in RAM, before the node OS starts. Syself Linux carries no cloud-init at all, and on bare metal the provisioner does the install the old script used to. Two things follow:

• It's more predictable. The steps are fixed and the environment is known, so there's little left that can vary between one build and the next.
• It reports back. Each step tells the controller what it's doing, so the platform can act on a stuck build instead of waiting on a node that never joins. That progress shows up on the machine object, readable with kubectl.

It also won't write a faulty node. A slow or unhealthy disk is caught before anything touches it, and a build that fails a check gets aborted instead of booted. So a broken machine never joins, and never turns into the dead, still-billed server you find weeks later. The full sequence is in the provisioning docs.

Where does the speed come from?

There's no build left to do at boot. No decompression, no file-by-file copy, no package install. The OS image is already written and verified, the control-plane container images already sit on local disk, and the slow Kubernetes setup steps already ran in the rescue system. At first boot the node only joins the cluster from the config staged for it.

Every node is built from the same image in the same known environment, and each build runs on its own machine. So a whole pool builds in parallel: ten nodes come up in about the time one takes.

What about GPU machines?

The image a node gets is matched to its hardware, and a GPU machine is the best example. Its NVIDIA driver isn't installed on the running node the way a general-purpose distribution would do it. It ships as another sealed layer the provisioner writes to disk in the rescue system, before first boot, alongside the base image. The driver is already built and signed against the exact kernel it will run under, so there's no compile step, no modprobe, and nothing to download once the node is up. A GPU node boots with a working driver already in place, and the device plugin that advertises its GPUs to the scheduler arrives as a managed in-cluster component. It's the same build-before-boot idea, extended to the one piece of hardware that usually forces a slow, fragile driver install after boot. The details are in Add GPU nodes.

One build, every job

The same build stands up a fresh cluster, replaces a failed node, and rolls a node during an upgrade. It runs the same way every time, so a throwaway cluster is cheap to spin up and discard, and a healed node is identical to one from a brand-new cluster. It's the same loop behind replacing nodes instead of patching them.

What are the limits?

The build runs inside the Hetzner rescue system, so if that's unavailable, new provisioning pauses. Running nodes keep working. A fresh extra control plane also waits up to about ten minutes for its cloud address before it can join, which applies to every control plane except the first one in a new cluster. And on first boot the kubelet restarts a few times until its config is written; that's expected and clears within the same boot, so watch node readiness rather than raw kubelet restarts.

FAQ

How long does a cluster take to provision?

About three minutes for a single node, and about nine for a full high-availability cluster of three control-plane and two worker nodes. These are typical figures, not guarantees.

Why build the node before it boots instead of after?

A live node that configures itself has a window where a step can fail or a network fetch can hang, and you end up with a half-built server the cluster already treats as real. Building it offline removes that window entirely.

Does provisioning pull anything from the network at boot?

No. The image, including the container images a control plane needs, is already on disk. A booting node downloads nothing, so a registry outage or blocked egress can't stall it.

Provisioning is how a node comes to life. What keeps it healthy afterward is node health and self-healing.