Skip to main content

Prepare workloads for upgrades

Inspect 1.36

An upgrade keeps the Kubernetes API and the control plane available the whole time. Your applications can stay up too, but only if you spread them across nodes and tell the cluster how much disruption they can take. The following pages apply the rules to stateless apps, to ingress and load balancers, and to databases.

To understand the rollout mechanics first, read .

What an upgrade does to your workload#

An upgrade replaces every node with a fresh, sealed machine. The control-plane side is the platform's job: control-plane nodes roll one at a time, etcd keeps its quorum, and the API server stays reachable throughout. You do not configure anything for the API to stay up.

The part that touches your workload is the worker side. Each worker node is replaced one at a time:

  1. The node is cordoned, so no new pods schedule onto it.
  2. The node is drained: every pod on it is evicted and has to reschedule onto another node.
  3. The node is deleted and a new node on the new version takes its place.

So an upgrade is a rolling node drain. A workload survives an upgrade exactly when it survives having its nodes drained one at a time. Everything here is about surviving that drain.

Why the defaults are not enough#

By default the cluster spreads pods across nodes with a soft rule (whenUnsatisfiable: ScheduleAnyway). Soft means best-effort: if the scheduler cannot honor the spread, it places the pod anyway rather than leaving it Pending. See for the default rules.

That default is convenient, but it does not protect a workload through a drain. Two replicas can land on the same node, either at first scheduling or after an earlier drain shuffled them there. When that one node is drained next, both replicas go down at once, and the workload has an outage even though it was running two replicas.

Consider a two-replica web app behind a load balancer:

  • With the default soft spread, both replicas can sit on the same worker. When that worker is replaced during the upgrade, the app is unreachable for as long as the replacement takes, often more than a minute.
  • With a hard spread and a PodDisruptionBudget, one replica always stays on another node while the first is drained. The app keeps serving through every node replacement, with only a few seconds of gap while the load balancer re-registers its targets on the new node.

The API server answered every request in both cases. The difference was entirely in how the workload was configured.

The four rules#

Every workload that must stay available needs all four. Missing any one of them reopens the outage.

# Rule Without it
1 Run at least two replicas A single replica is down for the whole time its node is replaced.
2 Spread the replicas across nodes with a hard rule, so no two share a node A drain can take two replicas down at once.
3 Set a PodDisruptionBudget the drain can satisfy within 180 seconds The drain gives up after nodeDrainTimeoutSeconds: 180 and the node is removed anyway, so a PDB nothing can satisfy just loses the pods.
4 Add a readiness probe and a graceful shutdown, so traffic reaches only ready pods and in-flight requests finish Traffic hits a pod that is starting or already shutting down, and requests fail.

Apply the rules to your workload type#

Related#

Migrate a manifest to v1beta2

Learn about the Cluster API v1beta2 update and how it affects your clusters.

Keep stateless workloads available

Configure replicas, a hard topology spread, a PodDisruptionBudget, readiness probes, and graceful shutdown so a stateless workload stays served through a Syself Autopilot upgrade.