A stateless workload stays available through an upgrade when it keeps at least one ready replica on a node that is not being drained. The [production-ready workload](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload) manifest already wires up the replicas, hard topology spread, PodDisruptionBudget, and readiness probe that make this hold. This page explains how each of those rules behaves while a node drains, and adds the graceful shutdown a stateless service needs to finish in-flight requests. Read [Prepare workloads for upgrades](/docs/hetzner/apalla/clusters/availability/prepare-workloads-for-upgrades) first for the context.

## Rule 1: run at least two replicas

A single replica has nowhere to go. When its node drains, the pod is evicted and the workload is down until it restarts on another node. Two is the minimum, so a second replica keeps serving while the first moves. Three or more leaves spare capacity during the move.

## Rule 2: spread the replicas across nodes with a hard rule

Two replicas do not help if they sit on the same node, because one drain takes both. The cluster's default spread is soft (`ScheduleAnyway`), which allows that. The manifest overrides it with a hard topology spread (`maxSkew: 1` over `kubernetes.io/hostname`, `whenUnsatisfiable: DoNotSchedule`), which keeps the replicas one-per-node until every node has one. A single drain then removes at most one replica.

Keep the replica count at or below your worker count for a strict one-per-node spread. Run more replicas than workers and some pods pause as `Pending` during a drain until the replacement node joins. That is fine as long as the budget below keeps enough replicas serving in the meantime.

Pod anti-affinity with `requiredDuringSchedulingIgnoredDuringExecution` on the same topology key is an equivalent hard rule. Use whichever your team already knows. See [Set affinity and anti-affinity](/docs/hetzner/apalla/workloads/placement/affinity-and-anti-affinity) for both in depth.

## Rule 3: set a PodDisruptionBudget

A drain evicts pods through the eviction API, which respects a PodDisruptionBudget. During a drain the budget holds the line: it evicts one pod, waits for the replacement to become ready on another node, then moves on, never taking the workload below the minimum you set. Without one, a drain can evict your last ready replica before its replacement is ready, and the workload has a gap.

Set the budget to leave the capacity your workload needs. The manifest uses `maxUnavailable`, which scales with the replica count; `minAvailable` expresses the same limit from the other side. A budget only helps if the workload can reschedule elsewhere, so `minAvailable` equal to the replica count blocks every eviction and stalls the drain. Keep at least one pod evictable.

## Rule 4: add a readiness probe and a graceful shutdown

Spreading and a budget keep a ready replica alive, but traffic still has to avoid pods that are starting or shutting down.

A **readiness probe** keeps a pod out of the Service endpoints until it can serve, so a new replica on a fresh node takes traffic only once it is ready. The manifest carries one; point it at your own health endpoint.

A **graceful shutdown** lets a pod that is being evicted finish its in-flight requests. When a pod is terminating, Kubernetes removes it from the Service endpoints and sends it `SIGTERM`, but a client or the load balancer may still send it a request for a moment. Give the pod time to drain:

```yaml
spec:
  template:
    spec:
      containers:
        - name: web
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]
      terminationGracePeriodSeconds: 30
```

The `preStop` sleep holds the pod for a few seconds after it leaves the endpoints, so in-flight requests land on a pod that is still serving. `terminationGracePeriodSeconds` is the ceiling for the whole shutdown. The container should also handle `SIGTERM` and stop accepting new work while it finishes what it has.

## The manifest

Start from the [production-ready workload](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload) manifest. It already carries Rules 1 through 4: the replicas, the hard topology spread, the PodDisruptionBudget, and the readiness probe. Add the graceful-shutdown block from Rule 4 for a stateless service that serves in-flight requests. Draining any single node then leaves ready replicas on the other nodes, and the drain waits for the moved replica to come back before the next node.

## Verify before you upgrade

- `kubectl get pods -n team-web -o wide -l app=web` shows the replicas on different nodes.
- `kubectl get pdb web -n team-web` shows `ALLOWED DISRUPTIONS` of at least 1. Zero means a drain cannot evict any pod and will stall.
- Draining one worker by hand (`kubectl drain <node> --ignore-daemonsets --delete-emptydir-data`) and watching the app stay served is the best way to check before you upgrade. Uncordon the node afterward.

## Related

- [Keep ingress and load balancers available](/docs/hetzner/apalla/clusters/availability/keep-ingress-and-load-balancers-available)
- [Keep databases available](/docs/hetzner/apalla/clusters/availability/keep-databases-available)
- [Steer workload placement](/docs/hetzner/apalla/workloads/placement/steer-workload-placement)