Keep workloads available
Keeping a workload available means keeping it served while its nodes are replaced. A node is replaced on a cluster upgrade, when self-healing removes an unhealthy node, or when you remove one yourself. In every case the pods on the node have to move. A workload stays up through that move only if you spread its pods across nodes and tell the cluster how much disruption it can take.
This page covers how a node replacement works and the conditions every workload needs. The pages below then apply those conditions to each workload type.
How a node is replaced
A node is always drained before it is removed, one node at a time:
- The node is cordoned, so no new pods schedule onto it.
- Its pods are evicted. The eviction waits on any PodDisruptionBudget that does not yet allow the pod to go.
- Each pod receives
SIGTERMand hasterminationGracePeriodSeconds(30 seconds by default) to shut down, then it is killed. - The pod reschedules onto another node and takes traffic once its readiness probe passes.
- When the node's pods are gone and their volumes are detached, the node is deleted.
DaemonSet and static pods are not evicted; they keep running until the node is gone.
A reboot is not a drain. When self-healing reboots a node, nothing is evicted and your PodDisruptionBudget is ignored: the pods go down with the node and come back with it. What helps there is having enough replicas on enough nodes. See Machine health checks and remediation .
Two timeouts bound the drain. Both are set per node pool:
| Setting | Default | When it is reached |
|---|---|---|
nodeDrainTimeoutSeconds | 180 seconds | The drain stops waiting. The node is deleted and any pods still on it go down with it. |
nodeVolumeDetachTimeoutSeconds | 120 seconds | The wait for volumes to detach ends and the deletion continues. |
A PodDisruptionBudget can slow a drain down, but not past nodeDrainTimeoutSeconds. Once that timeout is reached, the node is deleted with whatever is still running on it, without a graceful shutdown.
Verify worker node capacity
Cloud worker pools use maxSurge: 1: the stack creates the new node first and then removes the old one. Total capacity does not drop during a rollout.
Bare metal worker pools use maxSurge: 0: the stack removes an old node first and then provisions a new one. During a rollout, your pool runs one fewer worker node.
For bare metal pools, check that your workloads can tolerate one fewer node per pool during a rollout:
$ kubectl get nodes --selector='node.kubernetes.io/worker=true' -o wide
$ kubectl top nodes
If resource usage is high, either scale up the pool beforehand (temporarily) or wait for a lower-traffic period.
The conditions every workload needs
Every workload that must stay available through a node replacement needs these. They are the production baseline seen through the drain. The type pages below point back to this list.
| Condition | Without it |
|---|---|
| Run at least two replicas, and keep room on the other nodes for an evicted pod | A single replica is down for the whole move, or an evicted pod stays Pending. |
| Spread the replicas across nodes with a hard topology spread rule | One drain can take two replicas down at once. |
| Set a PodDisruptionBudget the drain can satisfy | The drain waits the full 180 seconds, then removes the node and loses the pods. |
| Add a readiness probe | Traffic reaches a pod before it can serve, and the drain moves on too early. |
Handle SIGTERM for a graceful shutdown | In-flight requests are cut off when the pod exits. |
| Keep capacity for one node at a time | Replacing a node leaves too few replicas serving. |
Choose your workload type
- Keep stateless workloads available : start here if you run a Deployment serving requests. The full manifest, PodDisruptionBudget sizing, and graceful shutdown.
- Keep ingress and load balancers available : start here if external traffic reaches you through an ingress controller behind a load balancer.
- Keep databases available : start here if you run a stateful database that has to keep its data and stay reachable.
Related
- Cluster upgrades : how a rollout replaces nodes, surge rules, and etcd quorum.
- Maintain a cluster : check cluster health before you upgrade.
- Self-healing and node replacement : when the platform replaces a node on its own.
- Steer workload placement : node selectors, affinity, and topology spread in depth.
Taints and tolerations
Add taints to node pools through the Cluster topology to control which pods can schedule on them.
Keep stateless workloads available
Apply the availability conditions to a stateless Deployment, size the PodDisruptionBudget and spread, and add the graceful shutdown that finishes in-flight requests through a node replacement.