Keep stateless workloads available

A stateless workload stays available through an upgrade when it keeps at least one ready replica on a node that is not being drained. The production-ready workload manifest already wires up the replicas, hard topology spread, PodDisruptionBudget, and readiness probe that make this hold. This page explains how each of those rules behaves while a node drains, and adds the graceful shutdown a stateless service needs to finish in-flight requests. Read Prepare workloads for upgrades first for the context.

Rule 1: run at least two replicas#

A single replica has nowhere to go. When its node drains, the pod is evicted and the workload is down until it restarts on another node. Two is the minimum, so a second replica keeps serving while the first moves. Three or more leaves spare capacity during the move.

Rule 2: spread the replicas across nodes with a hard rule#

Two replicas do not help if they sit on the same node, because one drain takes both. The cluster's default spread is soft (ScheduleAnyway), which allows that. The manifest overrides it with a hard topology spread (maxSkew: 1 over kubernetes.io/hostname, whenUnsatisfiable: DoNotSchedule), which keeps the replicas one-per-node until every node has one. A single drain then removes at most one replica.

Keep the replica count at or below your worker count for a strict one-per-node spread. Run more replicas than workers and some pods pause as Pending during a drain until the replacement node joins. That is fine as long as the budget below keeps enough replicas serving in the meantime.

Pod anti-affinity with requiredDuringSchedulingIgnoredDuringExecution on the same topology key is an equivalent hard rule. Use whichever your team already knows. See Set affinity and anti-affinity for both in depth.

Rule 3: set a PodDisruptionBudget#

A drain evicts pods through the eviction API, which respects a PodDisruptionBudget. During a drain the budget holds the line: it evicts one pod, waits for the replacement to become ready on another node, then moves on, never taking the workload below the minimum you set. Without one, a drain can evict your last ready replica before its replacement is ready, and the workload has a gap.

Set the budget to leave the capacity your workload needs. The manifest uses maxUnavailable, which scales with the replica count; minAvailable expresses the same limit from the other side. A budget only helps if the workload can reschedule elsewhere, so minAvailable equal to the replica count blocks every eviction and stalls the drain. Keep at least one pod evictable.

Rule 4: add a readiness probe and a graceful shutdown#

Spreading and a budget keep a ready replica alive, but traffic still has to avoid pods that are starting or shutting down.

A readiness probe keeps a pod out of the Service endpoints until it can serve, so a new replica on a fresh node takes traffic only once it is ready. The manifest carries one; point it at your own health endpoint.

A graceful shutdown lets a pod that is being evicted finish its in-flight requests. When a pod is terminating, Kubernetes removes it from the Service endpoints and sends it SIGTERM, but a client or the load balancer may still send it a request for a moment. Give the pod time to drain:

		spec:
  template:
    spec:
      containers:
        - name: web
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]
      terminationGracePeriodSeconds: 30
	

The preStop sleep holds the pod for a few seconds after it leaves the endpoints, so in-flight requests land on a pod that is still serving. terminationGracePeriodSeconds is the ceiling for the whole shutdown. The container should also handle SIGTERM and stop accepting new work while it finishes what it has.

The manifest#

Start from the production-ready workload manifest. It already carries Rules 1 through 4: the replicas, the hard topology spread, the PodDisruptionBudget, and the readiness probe. Add the graceful-shutdown block from Rule 4 for a stateless service that serves in-flight requests. Draining any single node then leaves ready replicas on the other nodes, and the drain waits for the moved replica to come back before the next node.

Verify before you upgrade#

• kubectl get pods -n team-web -o wide -l app=web shows the replicas on different nodes.
• kubectl get pdb web -n team-web shows ALLOWED DISRUPTIONS of at least 1. Zero means a drain cannot evict any pod and will stall.
• Draining one worker by hand (kubectl drain <node> --ignore-daemonsets --delete-emptydir-data) and watching the app stay served is the best way to check before you upgrade. Uncordon the node afterward.

Related#

• Keep ingress and load balancers available
• Keep databases available
• Steer workload placement