Run these checks before any cluster upgrade, whether you are bumping the cluster stack release within a minor version or moving to a new Kubernetes minor version. It is much easier to find a problem before the rollout starts than to diagnose it while the rollout is running.

## Prerequisites

- Access to the management cluster and the workload cluster (`kubectl` configured for both)
- `pluto` installed (static manifest scanner): [https://github.com/FairwindsOps/pluto](https://github.com/FairwindsOps/pluto)
- `kubent` installed (live cluster API scanner): [https://github.com/doitintl/kube-no-trouble](https://github.com/doitintl/kube-no-trouble)

## Check 1: Scan for deprecated and removed APIs

Kubernetes removes API versions in minor releases. If your workloads or Helm charts still reference a removed API version, they will stop working after the upgrade.

Run a static scan against your manifests and Helm releases:

```console
$ pluto detect-files -d ./manifests
$ pluto detect-helm
```

Run a live scan against the running cluster for objects using soon-to-be-removed APIs:

```console
$ kubent
```

Fix every finding before you proceed. Updating a manifest after the API is removed requires extra recovery steps.

## Check 2: Verify control plane health

The cluster must have a healthy control plane before a rollout starts. A three-node control plane needs at least two healthy etcd members to function. Do not start an upgrade if one control plane node is already unhealthy. During the upgrade that node gets replaced, so only one healthy etcd member would be left, and the cluster can lose quorum.

Check control plane node status:

```console
$ kubectl get nodes --selector='node-role.kubernetes.io/control-plane' -o wide
```

All control plane nodes must be `Ready`. If any are `NotReady` or show a problem condition, resolve that first.

## Check 3: Check for pending certificate signing requests

Pending CSRs (Certificate Signing Requests) can stall node join during a rollout. New nodes request certificates when they boot; if those requests are not approved quickly, the node takes longer to join or may time out.

List pending CSRs on the workload cluster (the cluster the new nodes are joining, not the management cluster):

```console
$ kubectl get csr
```

Any CSR in `Pending` state that has been waiting for more than a few minutes needs attention before you start a rollout. Approve legitimate ones or investigate why they are stalled.

## Check 4: Review PodDisruptionBudgets

The upgrade drain (moving all pods off a node before it is replaced) respects PodDisruptionBudgets (PDBs). A PDB limits how many pods of a group can be unavailable at once. If a PDB is configured too strictly, the drain stalls and the rollout waits.

List all PDBs in the workload cluster:

```console
$ kubectl get pdb -A
```

For each PDB, check:

- `ALLOWED DISRUPTIONS` is at least 1. If it is 0, the drain cannot proceed.
- The PDB actually has matched pods. A PDB with no matching pods causes no problem, but review it anyway.

```console
$ kubectl describe pdb -A
```

A PDB with `maxUnavailable: 0`, or `minAvailable` equal to the number of replicas, blocks the drain. When that happens, the system waits up to 180 seconds for the drain to finish, then gives up and moves on. This timeout is set in the cluster configuration (`nodeDrainTimeoutSeconds: 180`). After the timeout, the stack removes the node anyway, which can disrupt the pods that were still running on it. Adjust PDB settings to allow at least one pod to be unavailable per group, or increase replicas so the PDB can be satisfied during the drain.

> [!TIP]
> PDBs are the right tool for protecting availability during upgrades. Set `maxUnavailable: 1` (or a percentage that leaves at least one pod running) and run enough replicas that the PDB can be satisfied during a drain. A PDB with `maxUnavailable: 0` across all replicas blocks upgrades entirely.

## Check 5: Verify worker node capacity for bare metal pools

Cloud worker pools use `maxSurge: 1`: the stack creates the new node first, then removes the old one. Total capacity does not drop during the upgrade.

Bare metal worker pools use `maxSurge: 0`: the stack frees the old physical host first, then reprovisions it. During each step, your pool runs one fewer worker node.

For bare metal pools, check that your workloads can tolerate one fewer node per pool during the upgrade:

```console
$ kubectl get nodes --selector='node.kubernetes.io/worker=true' -o wide
$ kubectl top nodes
```

If resource usage is high, either scale up the pool before the upgrade (temporarily) or wait for a lower-traffic period.

## Check 6: Confirm all nodes are Ready

Start upgrades only when all nodes are `Ready`. An existing unhealthy node complicates rollout tracking and may trigger automatic node replacement during the upgrade.

```console
$ kubectl get nodes -o wide
```

If a node is `NotReady`, investigate and resolve the problem before starting the upgrade.

The checks below apply only when you move to a new Kubernetes minor version, and specifically to the 1.35 to 1.36 move. Skip them for a stack version bump within the same minor.

## Check 7: Remove ARM node pools before moving to 1.36

1.36 ships amd64 (x86-64) images only. It defines no `workerarm64hcloud` worker class and no arm64 (ARM) control plane, so a cluster that still runs ARM nodes cannot move to 1.36: there is no ARM image for the new nodes to boot.

Look for ARM pools and control planes in your `Cluster` object:

```console
$ kubectl get cluster <name> -o yaml | grep -i arm
```

A hit on `workerarm64hcloud` (a worker pool) or an `arm64` control-plane architecture variable means you must move those workloads onto x86 pools and remove the ARM pools before you upgrade.

## Check 8: Take a backup before you start

The 1.35 to 1.36 upgrade replaces the node operating system, and once the roll is under way it cannot be rolled back (see [Maintenance windows](#maintenance-windows) below). Take a backup first so you can rebuild if something goes wrong. Capture two layers:

- **GitOps state.** Confirm the cluster manifests and application manifests are committed and the repository is current. Re-syncing from Git is how you recover anything Git owns.
- **Runtime state and volume data.** Back up workloads and PersistentVolumeClaims that Git cannot reconstruct (data, hand-applied objects) with Velero.

See [Back up and restore](/docs/hetzner/apalla/storage/backup/back-up-with-velero).

## Check 9: Check your node ceiling if you use a custom pod CIDR

1.36 gives each node a `/24` pod subnet, about 254 usable addresses, which is what the per-node `maxPods: 220` cap is sized against. The most nodes a cluster can hold is set by how many bits sit between your pod CIDR prefix and `/24`:

```text
node ceiling = 2 ^ (24 − pod-CIDR prefix length)
```

The default pod CIDR is wide enough that this is not a concern (a `/11` pod CIDR holds `2 ^ (24 − 11) = 8192` nodes). A narrow custom pod CIDR lowers the ceiling: a `/16` holds `2 ^ (24 − 16) = 256` nodes, a `/18` holds 64. Check your pod CIDR:

```console
$ kubectl get cluster <name> -o jsonpath='{.spec.clusterNetwork.pods.cidrBlocks}'
```

If your node count is near the ceiling, plan your growth around it, or talk to us about widening the pod CIDR before you upgrade.

## Maintenance windows

There is no built-in maintenance window scheduler in Syself Autopilot. You control when an upgrade starts by controlling when you edit the `Cluster` object. Upgrades begin immediately when the spec change is applied.

Choose a time window when:

- Traffic load is low (less risk from temporary capacity reduction on bare metal pools)
- Your team is available to monitor the rollout and respond to problems
- No other changes are being applied to the cluster simultaneously

### How long the upgrade takes

Nodes roll one at a time, so total time scales with node count. Budget roughly 5 to 15 minutes per cloud node to provision, join, and drain the old node it replaces. Bare metal is slower: each host is released, wiped, and reinstalled, which runs to tens of minutes per node. The stack allows a cloud worker 10 minutes to come up (`nodeStartupTimeoutSeconds: 600`) and a control-plane or bare-metal worker node 30 minutes (`nodeStartupTimeoutSeconds: 1800`) before it treats the node as failed, so use that as the hard ceiling, not the typical case.

### The etcd window on a bare-metal control plane

A three-node control plane runs on three etcd members (the key-value store that holds cluster state), and it needs at least two to keep quorum. Bare-metal control planes use `maxSurge: 0`: each node is deleted and reprovisioned in place, not added alongside. So during each reprovision the control plane sits at two of three members. That still has quorum, but it has no spare. If a second member fails then, the control plane loses quorum and stops accepting changes. Do not run other risky operations (another node replacement, etcd maintenance, a network change) while a bare-metal control-plane node is reprovisioning.

### There is no rollback

Cluster API has no rollback, so every change is a roll forward. Plan an upgrade on the assumption that it has no undo. If a roll stalls, do not revert the spec: leave it in place, clear what is blocking it, and finish forward. [Stuck or failed upgrade](/docs/hetzner/apalla/clusters/troubleshooting/stuck-or-failed-upgrade) covers the recovery, and this is why the prechecks above matter.

> [!NOTE]
> If you use GitOps to manage your cluster (recommended), you can control the upgrade timing by controlling when the PR that changes the cluster's topology version is merged and applied.

## Related

- [Update within a version](/docs/hetzner/apalla/clusters/upgrades/update-within-a-version), patch and stack version bumps
- [Upgrade to a new Kubernetes version](/docs/hetzner/apalla/clusters/upgrades/upgrade-to-a-new-kubernetes-version), moving to the next minor version
- [Cluster upgrades](/docs/hetzner/apalla/concepts/operations/cluster-upgrades), rollout mechanics, surge rules, drain behavior, and PDB interplay
- [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement), how Syself detects and replaces unhealthy nodes during upgrades
- [Bare metal and cloud](/docs/hetzner/apalla/concepts/internals/bare-metal-and-cloud), the capacity and surge differences between backends