Skip to main content

Keep workloads available

Keeping a workload available means keeping it served while its nodes are replaced. A node is replaced on a cluster upgrade, when removes an unhealthy node, or when you remove one yourself. In every case the pods on the node have to move. A workload stays up through that move only if you spread its pods across nodes and tell the cluster how much disruption it can take.

This page covers how a node replacement works and the conditions every workload needs. The pages below then apply those conditions to each workload type.

How a node is replaced 

A node is always drained before it is removed, one node at a time:

  1. The node is cordoned, so no new pods schedule onto it.
  2. Its pods are evicted. The eviction waits on any PodDisruptionBudget that does not yet allow the pod to go.
  3. Each pod receives SIGTERM and has terminationGracePeriodSeconds (30 seconds by default) to shut down, then it is killed.
  4. The pod reschedules onto another node and takes traffic once its readiness probe passes.
  5. When the node's pods are gone and their volumes are detached, the node is deleted.

DaemonSet and static pods are not evicted; they keep running until the node is gone.

A reboot is not a drain. When self-healing reboots a node, nothing is evicted and your PodDisruptionBudget is ignored: the pods go down with the node and come back with it. What helps there is having enough replicas on enough nodes. See .

Two timeouts bound the drain. Both are set per node pool:

Setting Default When it is reached
nodeDrainTimeoutSeconds 180 seconds The drain stops waiting. The node is deleted and any pods still on it go down with it.
nodeVolumeDetachTimeoutSeconds 120 seconds The wait for volumes to detach ends and the deletion continues.

A PodDisruptionBudget can slow a drain down, but not past nodeDrainTimeoutSeconds. Once that timeout is reached, the node is deleted with whatever is still running on it, without a graceful shutdown.

Verify worker node capacity 

Cloud worker pools use maxSurge: 1: the stack creates the new node first and then removes the old one. Total capacity does not drop during a rollout.

Bare metal worker pools use maxSurge: 0: the stack removes an old node first and then provisions a new one. During a rollout, your pool runs one fewer worker node.

For bare metal pools, check that your workloads can tolerate one fewer node per pool during a rollout:

				$ kubectl get nodes --selector='node.kubernetes.io/worker=true' -o wide
$ kubectl top nodes

If resource usage is high, either scale up the pool beforehand (temporarily) or wait for a lower-traffic period.

The conditions every workload needs 

Every workload that must stay available through a node replacement needs these. They are the seen through the drain. The type pages below point back to this list.

Condition Without it
Run at least two replicas, and keep for an evicted pod A single replica is down for the whole move, or an evicted pod stays Pending.
Spread the replicas across nodes with a hard rule One drain can take two replicas down at once.
Set a the drain can satisfy The drain waits the full 180 seconds, then removes the node and loses the pods.
Add a Traffic reaches a pod before it can serve, and the drain moves on too early.
Handle SIGTERM for a In-flight requests are cut off when the pod exits.
Keep capacity for one node at a time Replacing a node leaves too few replicas serving.

Choose your workload type 

Related 

Taints and tolerations

Custom node taints don't survive here, so keep general workloads off dedicated nodes with a dedicated pool, a label, and a nodeSelector instead.

Keep stateless workloads available

Apply the availability conditions to a stateless Deployment, size the PodDisruptionBudget and spread, and add the graceful shutdown that finishes in-flight requests through a node replacement.