What's New in Syself Autopilot 1.36

The 1.36 release is the biggest change we've made to Syself Autopilot in a while. It brings Kubernetes 1.36 on a foundation we rebuilt from the node up.

Most of that work went into three things: the operating system running on your nodes, how those nodes get installed, and how they reach the control plane. All three used to lean on general-purpose tools that were never built for the way we run Kubernetes, so each one drifted over time, hid failures, or left a node hard to reason about. We replaced them with components we build and maintain ourselves, shaped around how the platform works. What you get is a foundation that is smaller, comes up faster, and can be checked down to the individual block on disk, with Kubernetes 1.36 running on top.

Your nodes now run Syself Linux. It is built from source and then sealed, so the kernel verifies every block on every read, and a node that has been tampered with is replaced instead of trusted. It is about 350 MB, ships no package manager, and carries nothing a Kubernetes node does not need. Before, nodes ran a general-purpose Linux we set up with scripts. That worked, but it was heavy, it drifted as it ran, and it was hard to prove what was on a given machine. Syself Linux closes that gap: the image is the same on every node and open to inspection, so what a machine is running is something you can verify rather than take on trust.

How a server gets installed is new too. The old path used cloud-init, or installimage on bare metal. Each one ran once on a booting machine we had no view into, and then it was done, so at scale you hit boot failures that no amount of testing caught, and the only symptom was a node that never joined the cluster. The new provisioner runs from the rescue system instead, and reports the real state of every step back to the platform as it happens, so a node that gets stuck tells us where it stopped rather than going silent. It has fewer moving parts, so there is less that can go wrong in the first place, and it is quicker: a node is ready about 60% faster than before.

The path between your nodes and the control plane changed as well. A worker now reaches the API server through a proxy on the node itself, not the Hetzner load balancer, so a load-balancer outage no longer knocks workers offline. A reverse tunnel carries the control plane's traffic back to your nodes for kubectl logs, exec, and metrics, over mTLS, since Kubernetes ships nothing to secure that path. We ran the konnectivity project here for five years and helped maintain its releases for the last year and a half. It is generic and barely maintained now, so we moved to our own, built for how we run. Every major cloud provider ended up making the same move. The last new piece is KubeGate, a passthrough proxy in front of every API server that preserves the real client IP and controls who reaches the control plane, so your audit logs finally name the real caller.

And Kubernetes itself moves to 1.36, bringing upstream features you can use in your workloads.

Everything else is below, grouped by area, with upgrade steps at the end.

At a glance

• Syself Linux: the node OS is built from source as sealed, integrity-checked layers, verified from boot and on every read.
• New provisioner: nodes install from the rescue system, and the platform watches provisioning and resolves problems instead of leaving a node that never joins.
• Nodes stay reachable when the load balancer is down: workers fall back to a direct control-plane connection instead of going NotReady.
• KubeGate: a new passthrough proxy in front of every API server. It preserves the real client IP and can restrict who reaches the control plane.
• Audit you can prove: four audit signals by default, including host auditd, the KubeGate audit, and the real client IP in the API server audit log.
• SSH can be fully blocked: it is out of the provisioning path entirely, and the host firewall is on from the first boot.
• Workloads: pick a runtime per pod, including a VM-isolated secure runtime, and schedule GPU pods with no setup.
• Prove what's inside: every component ships an SBOM and VEX advisories, tracked in the new Version Hub.
• Kubernetes 1.36: user namespaces, in-place pod resize, and more, plus Cilium 1.20 and the usual component updates.

How our releases work

We ship feature releases, and we bundle them with the Kubernetes minor versions. A new Kubernetes minor already brings new functionality and changes in behavior, so pairing our new features with it fits: change lands in one place, at one time, when you choose to move. Moving to a new Kubernetes minor is what we call an upgrade. You get new features, and some behavior can change. Staying on your current minor is an update, which brings security and bug fixes only and leaves the way the cluster behaves untouched. So the people who want stability update within a minor and keep the same behavior, and the people who want the newest features get them when they upgrade.

A feature release lands three to four times a year, one per Kubernetes minor version. Ours usually comes within about three months of that Kubernetes version shipping upstream. We need the window because we build on projects like Cilium, and they need time to ship a version that supports the new Kubernetes before we can build on top and test the whole thing together.

Within a minor version you still get a patched, secure system, and the Version Hub is how we keep that honest. It shows the full supply chain and the current CVE state of every component in a release. This is the release where that goes live: every component ships an SBOM you can download, alongside VEX advisories that tell you which CVEs actually affect you and which do not. We track what is moving in the security world there, so we can ship fixes fast and tell you what to do. See Supply chain security and Security evidence per release.

Syself Linux: the node OS, built from source and sealed

Your servers now run Syself Linux, an immutable OS we build from source as sealed, read-only layers, instead of packages installed onto a general-purpose base. We started building it over a year ago, and 1.36 is where it reaches every node. The base is verified from boot against a published root hash, and every cluster component and configuration has its integrity checked using dm-verity, ran block by block on every read, not only at boot. A modified base does not start, and tampering on a running node is caught and either replaces the node or raises an alert, depending on what was changed.

The sealed image cannot drift. Every server runs the same tested build, and an update is a fresh image, never a change applied in place. The base is about 350 MB and holds only what a Kubernetes node needs. It ships without a package manager or a general-purpose userspace, which leaves very little for an attacker to reach. We tune it for the hardware it runs on, and it ships with everything a node needs already cached, including the container images, so a node can come up even air-gapped. That is one less thing that can fail.

You can also prove what a node runs. Its root hash is recomputable from the published image, so anyone can confirm a node booted exactly what Syself shipped, no trust needed. Even a node's own settings are verifiable: the provisioner seals the per-node config in the rescue system as another dm-verity layer, so its hostname, keys, and network config are checked on every read like everything else. And you can browse what is on a node in the Version Hub: every component, its version, and a downloadable SBOM. See Supply chain security.

For more information, see Syself Linux, Node and OS security, and Verify node integrity.

A new provisioner installs each node

We rebuilt how a server gets installed. Provisioning used to run on cloud-init, in both cloud and bare metal clusters, plus Hetzner's installimage, a script that lays an operating system onto a bare metal server. A specialized Syself component now does all of that. The old tools each ran once on a booting node the platform could not see into, and then they were finished, so a failure at scale surfaced as a node that never joined the cluster, with little to go on. You can test to the 99th percentile and still hit the failures that only show up at scale.

The new provisioner runs from the Hetzner rescue system, before the node OS boots. It writes the sealed image to disk and stages the node's bootstrap data. Syself Linux carries no cloud-init at all, and on bare metal the provisioner does the install that Hetzner's installimage used to. Two things follow. It has fewer moving parts, since the steps are fixed and run in a known environment, so far less can vary and fail. And it reports the real state of each step back to the controller as it goes, so the platform knows what is actually happening on a node and can react, instead of waiting on a node that silently never joins. That progress shows on the machine object.

It is fast too. Getting a node ready is about 60% quicker than before, and that comes from two changes. The provisioner installs the OS about a quarter faster, and writing it now takes under a minute. Separately, we made the step that creates a server in Hetzner Cloud faster, which shrinks the wait before provisioning even starts. That part is provider-side, so clusters on older tracks get faster servers as well. The rest of a node's start-up is that server being created, which still differs between a cloud VM and a bare metal machine. A single node is ready in about three minutes, and a full HA cluster, three control-plane and two worker nodes, in about nine minutes.

Nodes stay reachable when the load balancer is down

Before, every worker reached the API server through a single control-plane load balancer. If that load balancer went down, every worker lost the API server at once, and after about 40 seconds Kubernetes marked them all NotReady and moved their pods. In 1.36 the kubelet no longer depends on that load balancer. It talks to a proxy on the node, and the proxy connects to the control planes and load-balances across them itself. A node stays connected even when the Hetzner load balancer, which is provider infrastructure outside our control, is unavailable, and the proxy routes around a control plane that is failing. It holds a long-lived HTTP/2 connection, so requests stay open and are multiplexed. This is a self-healing, client-side design in place of one shared dependency that could take every worker down at once.

The reverse tunnel is a separate piece, and it solves the opposite direction: how the API server reaches your nodes. kubectl logs, exec, and port-forward, and metrics from your pods, all need the API server to open a connection back to a node. Instead of the API server dialing in, each node dials out to the control plane and holds the connection open, and node-bound traffic travels back down it. Kubernetes ships no built-in way to secure this control-plane-to-node path, so the tunnel runs over mTLS: both ends prove their identity with a certificate, and the traffic is encrypted end to end. We ran konnectivity for five years, and for the last year and a half we helped maintain its releases, but its generic design never fit this job and it is barely maintained now. Every major cloud provider has walked away to its own fork or implementation, so we pivoted to our own tunnel, built for how we operate. That is why a new control plane is reachable the moment it boots, which removes the flapping that used to appear during control-plane scale-up; a dead connection reconnects in seconds; and in an HA setup the tunnel spreads node connections across control planes, so a busy or restarting one does not drag the others down.

KubeGate: unified, guarded access to the API server

KubeGate is a new component we built: a passthrough proxy that sits on every control-plane node in front of the API server. It takes the public API port, forwards the raw TLS to the API server on loopback, and never decrypts anything, so the API server still presents its own certificate and authenticates every caller itself. KubeGate is the single, controllable front door to the API server, and by design it can never lock a cluster node out. A node and the management plane always get through, and a joining node gets in with a short-lived signed ticket, before it even has a cluster identity.

You get real client attribution with no setup. The upstream Kubernetes API server cannot speak PROXY protocol, so behind a load balancer it would see the load balancer's address. KubeGate learns the real client address from the load balancer's PROXY-protocol header, then asks a privileged node daemon to open a real connection to the API server from that address and hand back the connected socket. Nothing rewrites a header or spoofs a packet: because the connection was genuinely made from the client's IP, the API server accepts that IP as the connection's true, unforgeable peer, even before you configure anything. On top of that, you can restrict which client IPs reach the API server through the load balancer to a list of trusted ranges, and your nodes and the platform keep working through it. The load balancer now checks each control plane with an HTTP /readyz probe answered by KubeGate, instead of a plain TCP port check, so it routes to a node only once that node's API server is actually serving.

See KubeGate: restrict API server access.

Audit you can prove

1.36 gives you four kinds of audit signal by default, ready to scrape.

• Host audit (auditd). Every node runs the Linux audit daemon with a fixed ruleset scoped for the BSI C5 cloud security standard, built from source and locked until reboot. It records who did what on the node.
• Kubernetes API audit. On control-plane nodes: who did what to which object through the API server. Because KubeGate passes the real client address through, an entry names the actual caller instead of the load balancer, even before you configure anything.
• KubeGate audit. The allow and deny decisions KubeGate makes, including denied requests, each recorded with its real origin.
• Tamper detection. The node's few writable paths, like logs, certificates, and kubelet state, are watched for changes that should not be there, such as an edited config or an added credential, and a change raises a node condition you can alert on. The rest of the OS is read-only, so a changed binary there is caught by the seal on the next read.

Nothing ships these off-node by default, so for compliance you retrieve them and send them to durable storage. See Retrieve audit logs, auditd rules, and Ship audit logs to a SIEM.

SSH and workload confinement

SSH is not needed while a node runs. It is a key-only door into the node for support and debugging, reachable from anywhere until you restrict it, so you can limit it to a bastion host or block the port entirely and nothing at runtime breaks. A future release removes SSH access by default. See Restrict SSH access.

Every non-privileged container runs under an enforcing AppArmor profile by default, with no setup. It blocks common container-breakout techniques while staying out of the way of ordinary workloads. Privileged pods run unconfined, so keep those rare. See Use AppArmor.

Guardrails for shared clusters

If you host several teams on one cluster, 1.36 adds admission-time guardrails that prevent common self-inflicted lockouts. A tenant webhook can no longer target the authentication, authorization, or admission APIs, which used to be a way to lock admins out or build a webhook loop. The platform's own admission policies protect themselves, so a tenant cannot remove them. See Control admission.

Network policy stays predictable on shared nodes. Cilium policy features that need a userspace proxy (HTTP-content rules, DNS rules, and toFQDNs) are rejected at apply time, so a policy fails with a readable error instead of silently breaking connectivity or DNS for other workloads. Plain address, port, and identity rules work as before. See Segment with network policies.

Workloads: runtimes and GPUs

Pick a runtime per pod. standard (crun) is the fast default and starts containers quicker on less memory. runc is the fallback for images that need it. secure runs the pod in its own lightweight VM on Kata Containers, so a container that escapes stays inside the VM; set runtimeClassName: secure and the pod lands on a bare metal node with hardware virtualization, or stays Pending if none exists rather than falling back to a shared kernel. See Workload runtimes and isolation and Run a secure workload.

GPU workloads schedule on their own. The NVIDIA device plugin now ships as a managed cluster component, so a GPU pod schedules with no manual setup once a supported GPU worker joins, and one card can back more than one pod through time-slicing. See Run GPU workloads.

Networking updates

Cilium moves to 1.20. Beyond the tunnel and KubeGate above, 1.36 tightens how load balancers and IP families behave, and where metrics listen.

• Serve IPv6 from an IPv4-only cluster. A type: LoadBalancer Service can still take IPv6 clients even though your nodes are IPv4-only. The Hetzner load balancer accepts the IPv6 connection and relays it to your IPv4 nodes. See Serve IPv6.
• Cleaner bare metal load balancers. In an IPv4-only cluster, the control-plane load balancer attaches only the node's IPv4 address, so there is no dead IPv6 backend showing unhealthy.
• bare metal provider IDs. bare metal nodes now use an hrobot:// provider ID. If you match on provider ID anywhere, read the upgrade note below.
• Metrics bind to loopback. Every in-cluster component that runs on the host network now exposes its metrics on 127.0.0.1, so unauthenticated metrics are never reachable on a node's public IP.

Clearer signals from your nodes

A new version of syself-health runs on every node. It reports problems as structured conditions on the Node object, and on bare metal it now also tracks the server's storage, so a failing disk shows up as node data you can alert on through your observability stack. See Self-healing and node replacement and node health conditions.

There is also a new log collector. It gathers what Syself needs to diagnose a problem in one step, so opening a support case is quick instead of a hunt for the right logs. See Collect logs with the log collector.

Kubernetes 1.36

Every cluster on the track runs Kubernetes 1.36.3. Some upstream features you can use right away:

• In-place pod resizing. Adjust a pod's CPU and memory budget without a restart: no redeploy, no dropped connections.
• Pod-level resource budgets. Set requests and limits once for the whole pod, so sidecars and the main container share headroom instead of each reserving their own.
• User namespaces, now stable. One field in your pod spec maps container root to an unprivileged host user, so a breakout gains nothing.
• Resize jobs before they run. Update the CPU and memory requests on a suspended Job, then unsuspend it. No more delete-and-recreate to fit your quota.
• Unprivileged container builds. ProcMount goes stable, so combined with user namespaces, nested container workloads no longer need a privileged pod.
• Hardware faults in pod status. Device health surfaces directly in a pod's status, separating a failing device from a failing container image.

For the full upstream list, see the Kubernetes 1.36 release notes.

The platform pins a patch version that moves forward as Kubernetes ships security fixes. Check your manifests against the 1.36 API removals before you upgrade. See Kubernetes version support.

Release notes by area

The full list, grouped. The sections above cover the big items; this is the map.

• Security: sealed from-source node OS, four audit signals (auditd, API audit, KubeGate audit, tamper detection), SSH blockable at runtime, AppArmor enforcing by default, tenant admission guardrails, proxy-based network-policy features rejected, per-release SBOM and VEX in the Version Hub.
• Networking: node-local failover proxy with client-side load balancing, mTLS reverse tunnel replacing konnectivity, KubeGate with an HTTP /readyz load-balancer probe, IPv6 Services from IPv4-only clusters, hrobot:// bare metal provider IDs, Cilium 1.20, Hubble relay metrics.
• Compute: standard, runc, and secure (Kata) runtimes per pod, NVIDIA device plugin with GPU time-slicing.
• Platform: new rescue-system provisioner, Kubernetes 1.36.3, and updated core components (etcd, CoreDNS, the Hetzner Cloud Controller Manager).
• Storage: CSI controller metrics for provisioning, attach, and resize.
• Observability: host-network metrics bound to 127.0.0.1, CSI and Hubble relay metrics, syself-health node conditions and bare metal storage, new log collector.
• Lifecycle: ordered 1.35 to 1.36 upgrade that keeps the API and workloads available, with node-image-dependent addons (the GPU plugin, the runtime classes) applied only after every node has rolled.

Before you upgrade

• Kubernetes API removals. Check your manifests against the 1.36 API removals.
• Metrics moved to loopback. If you scrape host-network component metrics from off the node, those endpoints now listen on 127.0.0.1 and refuse an off-node connection. Move to a host-network collector on the node. See Scrape control-plane metrics.
• Stricter node and confinement. The node OS is read-only and a non-privileged container is AppArmor-confined by default. A workload that needs extra Linux privileges must request them in its securityContext. A custom node agent that writes to the host or expects to run unconfined can break; test it in a separate cluster first and reach out for help.
• Network policies. Confirm your policies use address, port, or identity rules. HTTP-content and DNS-based rules are rejected on this platform.
• bare metal provider IDs. bare metal nodes move to an hrobot:// provider ID. Update anything that matches on the old provider-ID form.
• Single-stack IPv4. 1.36 clusters are IPv4-only. You can still serve a Service to IPv6 clients through the load balancer.