Keep stateless workloads available

A stateless workload stays available through a node replacement when it keeps at least one ready replica on a node that is not being drained. The conditions every workload needs explain what each setting does and why a drain needs it. This page applies them to a Deployment, sizes the budget and spread, and adds the graceful shutdown a stateless service needs to finish in-flight requests. The manifest in Survive node drains already wires up the replicas, hard topology spread, PodDisruptionBudget, and readiness probe.

Why the defaults are not enough 

By default the cluster spreads pods across nodes with a soft rule (whenUnsatisfiable: ScheduleAnyway). Soft means best-effort: if the scheduler cannot honor the spread, it places the pod anyway rather than leaving it Pending. See Steer workload placement for the default rules.

That default is convenient, but it does not protect a workload through a drain. Two replicas can land on the same node, either at first scheduling or after an earlier drain shuffled them there. When that one node is drained next, both replicas go down at once, and the workload has an outage even though it was running two replicas.

Consider a two-replica web app behind a load balancer:

• With the default soft spread, both replicas can sit on the same worker. When that worker is replaced, the app is unreachable for as long as the replacement takes, often more than a minute.
• With a hard spread and a PodDisruptionBudget, one replica always stays on another node while the first is drained. The app keeps serving through every node replacement, with only a few seconds of gap while the load balancer re-registers its targets on the new node.

The difference is entirely in how the workload was configured.

Size the spread and budget for a Deployment 

Use a hard topology spread (maxSkew: 1 over kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule), which keeps the replicas one-per-node until every node has one, so a single drain removes at most one replica. Keep the replica count at or below your worker count for a strict one-per-node spread. Run more replicas than workers and some pods pause as Pending during a drain until the replacement node joins; that is fine as long as the budget keeps enough replicas serving in the meantime. Pod anti-affinity with requiredDuringSchedulingIgnoredDuringExecution on the same topology key is an equivalent hard rule; use whichever your team already knows. See Set affinity and anti-affinity .

Size the PodDisruptionBudget to leave the capacity your workload needs. The manifest uses maxUnavailable, which scales with the replica count; minAvailable expresses the same limit from the other side. A budget only helps if the workload can reschedule elsewhere, so minAvailable equal to the replica count blocks every eviction and stalls the drain. Keep at least one pod evictable.

Add a graceful shutdown 

A graceful shutdown lets a pod that is being evicted finish its in-flight requests. When a pod is terminating, Kubernetes removes it from the Service endpoints and sends it SIGTERM, but a client or the load balancer may still send it a request for a moment. Give the pod time to drain:

				spec:
  template:
    spec:
      containers:
        - name: web
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]
      terminationGracePeriodSeconds: 30
			

The preStop sleep holds the pod for a few seconds after it leaves the endpoints, so in-flight requests land on a pod that is still serving. terminationGracePeriodSeconds is the ceiling for the whole shutdown. The container should also handle SIGTERM and stop accepting new work while it finishes what it has.

The manifest 

Start from the manifest in Survive node drains . It already carries the replicas, the hard topology spread, the PodDisruptionBudget, and the readiness probe. Add the graceful-shutdown block above for a stateless service that serves in-flight requests. Draining any single node then leaves ready replicas on the other nodes, and the drain waits for the moved replica to come back before the next node.

Verify before you upgrade 

• kubectl get pods -n team-web -o wide -l app=web shows the replicas on different nodes.
• kubectl get pdb web -n team-web shows ALLOWED DISRUPTIONS of at least 1. Zero means a drain cannot evict any pod and will stall.
• Draining one worker by hand (kubectl drain <node> --ignore-daemonsets --delete-emptydir-data) and watching the app stay served is the best way to check before you upgrade. Uncordon the node afterward.

Related 

• Keep workloads available : how a node replacement works and the conditions every workload needs.
• Keep ingress and load balancers available
• Keep databases available
• Steer workload placement