Keeping a workload available means keeping it served while its nodes are replaced. A node is replaced on a cluster upgrade, when [self-healing](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement) removes an unhealthy node, or when you remove one yourself. In every case the pods on the node have to move. A workload stays up through that move only if you spread its pods across nodes and tell the cluster how much disruption it can take.

This page covers how a node replacement works and the conditions every workload needs. The pages below then apply those conditions to each workload type.

## How a node is replaced

A node is always drained before it is removed, one node at a time:

1. The node is **cordoned**, so no new pods schedule onto it.
2. Its pods are **evicted**. The eviction waits on any PodDisruptionBudget that does not yet allow the pod to go.
3. Each pod receives `SIGTERM` and has `terminationGracePeriodSeconds` (30 seconds by default) to shut down, then it is killed.
4. The pod reschedules onto another node and takes traffic once its readiness probe passes.
5. When the node's pods are gone and their volumes are detached, the node is deleted.

DaemonSet and static pods are not evicted; they keep running until the node is gone.

A reboot is not a drain. When self-healing reboots a node, nothing is evicted and your PodDisruptionBudget is ignored: the pods go down with the node and come back with it. What helps there is having enough replicas on enough nodes. See [Machine health checks and remediation](/docs/hetzner/apalla/servers-and-nodes/maintenance/machine-health-checks-and-remediation).

Two timeouts bound the drain. Both are set per node pool:

| Setting                          | Default     | When it is reached                                                                     |
| -------------------------------- | ----------- | -------------------------------------------------------------------------------------- |
| `nodeDrainTimeoutSeconds`        | 180 seconds | The drain stops waiting. The node is deleted and any pods still on it go down with it. |
| `nodeVolumeDetachTimeoutSeconds` | 120 seconds | The wait for volumes to detach ends and the deletion continues.                        |

A PodDisruptionBudget can slow a drain down, but not past `nodeDrainTimeoutSeconds`. Once that timeout is reached, the node is deleted with whatever is still running on it, without a graceful shutdown.

### Verify worker node capacity

Cloud worker pools use `maxSurge: 1`: the stack creates the new node first and then removes the old one. Total capacity does not drop during a rollout.

Bare metal worker pools use `maxSurge: 0`: the stack removes an old node first and then provisions a new one. During a rollout, your pool runs one fewer worker node.

For bare metal pools, check that your workloads can tolerate one fewer node per pool during a rollout:

```console
$ kubectl get nodes --selector='node.kubernetes.io/worker=true' -o wide
$ kubectl top nodes
```

If resource usage is high, either scale up the pool beforehand (temporarily) or wait for a lower-traffic period.

## The conditions every workload needs

Every workload that must stay available through a node replacement needs these. They are the [production baseline](/docs/hetzner/apalla/workloads/production/survive-node-drains) seen through the drain. The type pages below point back to this list.

| Condition                                                                                                                                                | Without it                                                                      |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Run at least two replicas, and keep [room on the other nodes](/docs/hetzner/apalla/workloads/production/resource-requests-and-limits) for an evicted pod | A single replica is down for the whole move, or an evicted pod stays `Pending`. |
| Spread the replicas across nodes with a hard [topology spread](/docs/hetzner/apalla/workloads/placement/topology-spread-constraints) rule                | One drain can take two replicas down at once.                                   |
| Set a [PodDisruptionBudget](/docs/hetzner/apalla/workloads/production/pod-disruption-budgets) the drain can satisfy                                      | The drain waits the full 180 seconds, then removes the node and loses the pods. |
| Add a [readiness probe](/docs/hetzner/apalla/workloads/production/health-probes)                                                                         | Traffic reaches a pod before it can serve, and the drain moves on too early.    |
| Handle `SIGTERM` for a [graceful shutdown](/docs/hetzner/apalla/workloads/production/graceful-shutdown)                                                  | In-flight requests are cut off when the pod exits.                              |
| Keep capacity for one node at a time                                                                                                                     | Replacing a node leaves too few replicas serving.                               |

## Choose your workload type

- [Keep stateless workloads available](/docs/hetzner/apalla/workloads/availability/keep-stateless-workloads-available): start here if you run a Deployment serving requests. The full manifest, PodDisruptionBudget sizing, and graceful shutdown.
- [Keep ingress and load balancers available](/docs/hetzner/apalla/workloads/availability/keep-ingress-and-load-balancers-available): start here if external traffic reaches you through an ingress controller behind a load balancer.
- [Keep databases available](/docs/hetzner/apalla/workloads/availability/keep-databases-available): start here if you run a stateful database that has to keep its data and stay reachable.

## Related

- [Cluster upgrades](/docs/hetzner/apalla/concepts/operations/cluster-upgrades): how a rollout replaces nodes, surge rules, and etcd quorum.
- [Maintain a cluster](/docs/hetzner/apalla/clusters/upgrades/maintain-a-cluster): check cluster health before you upgrade.
- [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement): when the platform replaces a node on its own.
- [Steer workload placement](/docs/hetzner/apalla/workloads/placement/steer-workload-placement): node selectors, affinity, and topology spread in depth.