Documentation is tailored to each Cluster Stack release. Pick the one your cluster runs. {#if release.hasArmImages && minor !== stack.latest.minor} ## Check for ARM-based nodes If your cluster isn't running ARM-based nodes, skip this step. ARM is temporarily not supported on this release. If your cluster has any, the upgrade will fail unless you migrate them to x86 nodes first — see [Migrate ARM servers to x86 in your cluster](/docs/hetzner/apalla/servers-and-nodes/provision/migrate-arm-to-x86) for instructions. {/if} {#if minor === stack.latest.minor} ## You are here You are seeing the documentation for Kubernetes 1.35 — the newest Kubernetes minor Syself Autopilot supports right now. If your cluster is already on this version, there's nothing to do. Not on this version? Pick your cluster's actual version in the picker above to jump straight to the guide that applies to it. ## Not sure which Kubernetes version your cluster is on? If you're not sure which Kubernetes version your cluster is currently on, check it directly: Check in your cluster Run these commands against the Syself Autopilot management cluster, not the workload cluster. To check one cluster: ```console $ kubectl get cluster -n \ -o jsonpath='{.spec.topology.classRef.name}{"\n"}' ``` The command returns the Cluster Stack release name: ```console vars hetzner-apalla-1-35-v4 ``` The name includes the provider, stack, Kubernetes minor version, and Cluster Stack release. The final `vN` segment identifies the Cluster Stack release. To check every cluster: ```console $ kubectl get clusters -A \ -o custom-columns='NAMESPACE:.metadata.namespace,CLUSTER:.metadata.name,CLUSTERSTACK:.spec.topology.classRef.name,KUBERNETES:.spec.topology.version' ``` **If the management cluster serves `cluster.x-k8s.io/v1beta1`**: If the Syself Autopilot management cluster serves `cluster.x-k8s.io/v1beta1` instead of `v1beta2`, use `spec.topology.class` instead of `spec.topology.classRef.name`. To check one cluster with `cluster.x-k8s.io/v1beta1`: ```console $ kubectl get cluster -n \ -o jsonpath='{.spec.topology.class}{"\n"}' ``` To check every cluster with `cluster.x-k8s.io/v1beta1`: ```console $ kubectl get clusters -A \ -o custom-columns='NAMESPACE:.metadata.namespace,CLUSTER:.metadata.name,CLUSTERSTACK:.spec.topology.class,KUBERNETES:.spec.topology.version' ``` If it doesn't match, change your version in the version picker If it doesn't show 1.35, switch the docs to your cluster's actual version using the picker below: An upgrade guide scoped to that version will walk you through the upgrade path to get here. ## Explore further Exactly which components and versions hetzner-apalla-1-36-v2 ships. Every maintained Kubernetes minor and Cluster Stack release, including 1.36's full history, with support windows and what changed between them. ## Already on this Kubernetes version but want the latest Cluster Stack release for it? See [Updating the cluster with same Kubernetes version](/docs/hetzner/apalla/clusters/upgrades/update-within-a-version) to get the latest Cluster Stack release for Kubernetes 1.35. {:else if upgrade.toNextMinor.length === 0 || upgrade.toNextMinor[0] !== upgrade.nextMinorEntry} You are viewing documentation for Kubernetes 1.35, Cluster Stack release hetzner-apalla-1-35-v4. To upgrade to Kubernetes 1.36, first update your cluster to Cluster Stack release hetzner-apalla-1-35-v4 by following this guide: [Updating the cluster with same Kubernetes version](/docs/hetzner/apalla/clusters/upgrades/update-within-a-version). Once that update is complete, return to this guide to upgrade your Kubernetes version. Not sure which Kubernetes minor and Cluster Stack release your cluster uses? [Check the Kubernetes version and Cluster Stack](/docs/hetzner/apalla/clusters/configure/how-to-check-the-kubernetes-version-and-stack) of your cluster. {:else} {#if nextMinor?.version !== '1.36'} This guide will detail the process of upgrading your cluster from Kubernetes 1.35 to 1.36. Not sure which Kubernetes minor and Cluster Stack release your cluster uses? [Check the Kubernetes version and Cluster Stack](/docs/hetzner/apalla/clusters/configure/how-to-check-the-kubernetes-version-and-stack) of your cluster. {/if} {#if nextMinor?.version === '1.36'} ## Before you upgrade to 1.36 Upgrading to 1.36 moves your nodes to a new operating system, Syself Linux, and makes a few defaults stricter. As in any upgrade, every node is replaced, and your workloads keep running through it. Most of them need no changes. The ones that might are below, with what to check for each. For everything else that changed in 1.36, see [What's New in Syself Autopilot 1.36](/resources/whats-new-syself-autopilot-1-36). ### If you run something on the node itself Syself Linux is immutable and minimal: no package manager, and nothing on it a Kubernetes node doesn't need. A DaemonSet still runs fine. What breaks is installing a binary onto the host, writing to it, or loading a kernel module — none of which was ever officially supported, though it worked on the old general-purpose Linux. We call it out only in case you relied on it; for most clusters we expect no problems here. If you did do it — mostly monitoring agents, security agents, and storage helpers — check that each one works on a read-only host, and test it in a non-production cluster first. ### Known breaking changes Most workloads run unchanged. If one of these describes your cluster, change it before upgrading. - Pods that need a syscall like `perf_event_open`, `ptrace`, `mount`, `unshare`, or `bpf` (profilers, debuggers) fail. - Pods with a `hostPath` that writes to the OS disk break, because the OS disk is now read-only. Only `/var` stays writable, so a `hostPath` under `/var` is fine; any path outside it is not. Nodes customized over SSH break for the same reason. - In-kernel CIFS/SMB mounts, NFS exported from the node kernel, or custom kernel modules stop working. - A DaemonSet or agent that shells out to `iptables` or `nftables` on the host fails. - A self-managed GPU device plugin or gpu-operator conflicts with the managed one. ### AppArmor Every non-privileged container now runs under an enforcing AppArmor profile, with nothing to set up. It leaves normal workloads alone and blocks what a container rarely does: writing to host paths, opening raw sockets, tracing other processes. A workload that needs one of those keeps working once it declares it in `securityContext`. Privileged pods run unconfined, so keep those rare. See [Use AppArmor](/docs/hetzner/apalla/security/use-apparmor) for how to allow what a workload needs. ### Admission webhooks A validating or mutating webhook can no longer point at the authentication, authorization, or admission APIs. That was a way to lock admins out or build a webhook loop, so 1.36 rejects any webhook scoped that wide. If you run your own, scope each one to the resources it inspects. Syself's own admission policies protect themselves, so nothing removes them by accident. ### Host-network metrics Every component on the host network now serves its metrics on `127.0.0.1`, so nothing is exposed on a node's public IP. A Prometheus or agent that scrapes control-plane or other host-network metrics from outside the node stops getting them. Move that scraping onto the node, to a collector on the host network. See [Control-plane metrics](/docs/hetzner/apalla/observability/metrics/control-plane-metrics) for the supported way to collect them. ### Network policies Cilium moves to 1.20, and policies that need its userspace proxy (HTTP rules, DNS rules, and `toFQDNs`) are now [rejected when you apply them](/docs/hetzner/apalla/security/segment-with-network-policies), with a clear error instead of quietly breaking connectivity for other workloads on the node. Plain address, port, and identity rules work as before. If any of your policies use the proxy features, rewrite them before you upgrade. ## Ready to upgrade This guide will detail the process of upgrading your cluster from Kubernetes 1.35 to 1.36. If you're not sure any of the changes above affect you, create a new cluster on 1.36 first, deploy your workloads there, and confirm they run before you upgrade a running cluster. Not sure which Kubernetes minor and ClusterStack release your cluster uses? [Check the Kubernetes version and Cluster Stack](/docs/hetzner/apalla/clusters/configure/how-to-check-the-kubernetes-version-and-stack) of your cluster. {/if} ## Editing the cluster object {#if upgrade.nextMinorEntry === nextMinor?.latest} Change your cluster's `spec.topology.classRef.name` to hetzner-apalla-1-36-v2 and `spec.topology.version` to v1.36.3, as shown below. Both belong to the same release — set them together and you're done. ```yaml vars persistBlur title="cluster.yaml" ... name: mycluster spec: topology: classRef: name: hetzner-apalla-1-36-v2 version: v1.36.3 controlPlane: replicas: 3 workers: machineDeployments: - class: workeramd64hcloud name: md-0 ... ``` {:else} This upgrade spans several releases. Change your cluster's `spec.topology.classRef.name` and `spec.topology.version` one release at a time, following the steps below. {/if} This can be achieved either by editing the resource inside the management cluster, or by updating it in your GitOps repository. Once you apply it, the cluster rolls to the new Kubernetes minor on its own: new nodes come up and the old ones are removed, one at a time. This is the same node replacement that happens during routine self-healing, so a workload set up to tolerate it — replicas spread across nodes with a `PodDisruptionBudget` — rides through the roll without downtime. ## Confirm the upgrade finished When the roll completes, check that every node reports the new Kubernetes version and `Ready`, and that your workloads still serve traffic: ```console $ kubectl get nodes ``` {#if nextMinor?.version === '1.36'} ## Debugging a node changes You cannot install tools on a node. Use `kubectl debug node/` to get a debug pod with host access, or run a privileged pod with the tools you need. SSH still reaches the node on port 100 for operators, but the node ships no package manager to add tools there. For the full node-debugging workflow, see [Emergency SSH access](/docs/hetzner/apalla/servers-and-nodes/access/emergency-ssh-access). ## Rollback and the point of no return As always with Cluster API, once you apply the version change the upgrade rolls out to the end — there is no rollback option. Before you apply it, reverting the version in the `Cluster` spec is fine. After that, the only direction is forward: if a roll stalls, leave the spec in place and contact support. {/if} {/if} > [!NOTE] > Starting from hetzner-apalla-1-35-v1, Hetzner node image builds are no longer required. Because of that, we require a common SSH key for both bare metal and cloud servers. If you did not use the same SSH key when doing the [Hetzner account preparation](/docs/hetzner/apalla/getting-started/prepare-your-hetzner-account), please change the secrets to ensure you have a single SSH key.