Upgrading the cluster from Kubernetes 1.35 to Kubernetes 1.36
Documentation is tailored to each Cluster Stack release. Pick the one your cluster runs.
Before you upgrade to 1.36#
Upgrading to 1.36 moves your nodes to a new operating system, Syself Linux, and makes a few defaults stricter. As in any upgrade, every node is replaced, and your workloads keep running through it. Most of them need no changes. The ones that might are below, with what to check for each. For everything else that changed in 1.36, see What's New in Syself Autopilot 1.36.
If you run something on the node itself#
Syself Linux is immutable and minimal: no package manager, and nothing on it a Kubernetes node doesn't need. A DaemonSet still runs fine. What breaks is installing a binary onto the host, writing to it, or loading a kernel module — none of which was ever officially supported, though it worked on the old general-purpose Linux. We call it out only in case you relied on it; for most clusters we expect no problems here. If you did do it — mostly monitoring agents, security agents, and storage helpers — check that each one works on a read-only host, and test it in a non-production cluster first.
Known breaking changes#
Most workloads run unchanged. If one of these describes your cluster, change it before upgrading.
- Pods that need a syscall like
perf_event_open,ptrace,mount,unshare, orbpf(profilers, debuggers) fail. - Pods with a
hostPaththat writes to the OS disk break, because the OS disk is now read-only. Only/varstays writable, so ahostPathunder/varis fine; any path outside it is not. Nodes customized over SSH break for the same reason. - In-kernel CIFS/SMB mounts, NFS exported from the node kernel, or custom kernel modules stop working.
- A DaemonSet or agent that shells out to
iptablesornftableson the host fails. - A self-managed GPU device plugin or gpu-operator conflicts with the managed one.
AppArmor#
Every non-privileged container now runs under an enforcing AppArmor profile, with nothing to set up. It leaves normal workloads alone and blocks what a container rarely does: writing to host paths, opening raw sockets, tracing other processes. A workload that needs one of those keeps working once it declares it in securityContext. Privileged pods run unconfined, so keep those rare. See Use AppArmor for how to allow what a workload needs.
Admission webhooks#
A validating or mutating webhook can no longer point at the authentication, authorization, or admission APIs. That was a way to lock admins out or build a webhook loop, so 1.36 rejects any webhook scoped that wide. If you run your own, scope each one to the resources it inspects. Syself's own admission policies protect themselves, so nothing removes them by accident.
Host-network metrics#
Every component on the host network now serves its metrics on 127.0.0.1, so nothing is exposed on a node's public IP. A Prometheus or agent that scrapes control-plane or other host-network metrics from outside the node stops getting them. Move that scraping onto the node, to a collector on the host network. See Control-plane metrics for the supported way to collect them.
Network policies#
Cilium moves to 1.20, and policies that need its userspace proxy (HTTP rules, DNS rules, and toFQDNs) are now rejected when you apply them , with a clear error instead of quietly breaking connectivity for other workloads on the node. Plain address, port, and identity rules work as before. If any of your policies use the proxy features, rewrite them before you upgrade.
Ready to upgrade#
This guide will detail the process of upgrading your cluster from Kubernetes 1.35 to 1.36.
Unsure how your workloads will behave?
If you're not sure any of the changes above affect you, create a new cluster on 1.36 first, deploy your workloads there, and confirm they run before you upgrade a running cluster.
Not sure which Kubernetes minor and ClusterStack release your cluster uses? Check the Kubernetes version and Cluster Stack of your cluster.
Editing the cluster object#
Change your cluster's spec.topology.classRef.name to hetzner-apalla-1-36-v2 and spec.topology.version to v1.36.3, as shown below. Both belong to the same release — set them together and you're done.
...
name: mycluster
spec:
topology:
classRef:
name: hetzner-apalla-1-36-v2
version: v1.36.3
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: workeramd64hcloud
name: md-0
...
This can be achieved either by editing the resource inside the management cluster, or by updating it in your GitOps repository.
Once you apply it, the cluster rolls to the new Kubernetes minor on its own: new nodes come up and the old ones are removed, one at a time. This is the same node replacement that happens during routine self-healing, so a workload set up to tolerate it — replicas spread across nodes with a PodDisruptionBudget — rides through the roll without downtime.
Confirm the upgrade finished#
When the roll completes, check that every node reports the new Kubernetes version and Ready, and that your workloads still serve traffic:
$ kubectl get nodes
Debugging a node changes#
You cannot install tools on a node. Use kubectl debug node/<node> to get a debug pod with host access, or run a privileged pod with the tools you need. SSH still reaches the node on port 100 for operators, but the node ships no package manager to add tools there. For the full node-debugging workflow, see Emergency SSH access .
Rollback and the point of no return#
As always with Cluster API, once you apply the version change the upgrade rolls out to the end — there is no rollback option. Before you apply it, reverting the version in the Cluster spec is fine. After that, the only direction is forward: if a roll stalls, leave the spec in place and contact support.
Note
Starting from hetzner-apalla-1-35-v1, Hetzner node image builds are no longer required. Because of that, we require a common SSH key for both bare metal and cloud servers. If you did not use the same SSH key when doing the Hetzner account preparation , please change the secrets to ensure you have a single SSH key.