Syself Autopilot drains and replaces nodes often, on every rolling upgrade and whenever [self-healing](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement) replaces an unhealthy node. A bare Deployment loses traffic on each of those. This is the one page that lists every setting that keeps it serving. Apply all of them for anything serving real traffic; each has its own page for the detail. For how these rules behave during a drain and the graceful shutdown a stateless service adds, see [Keep stateless workloads available](/docs/hetzner/apalla/clusters/availability/keep-stateless-workloads-available).

## The baseline

Five settings, none optional for real traffic. Miss one and the others cannot cover for it.

- **At least two replicas.** A single replica goes fully down the moment its node drains. A second keeps serving while the first moves.
- **A readiness probe** (and usually a liveness probe), so traffic only reaches a pod that can serve, and a stuck process gets restarted. See [Configure health probes](/docs/hetzner/apalla/workloads/production/health-probes).
- **Resource requests on every container.** A request-less pod looks free to the scheduler and the autoscaler, so it gets packed onto a full node and evicted under pressure. See [Set resource requests and limits](/docs/hetzner/apalla/workloads/production/resource-requests-and-limits).
- **A PodDisruptionBudget**, so a drain waits rather than take your last pod down. See [Add a PodDisruptionBudget](/docs/hetzner/apalla/workloads/production/pod-disruption-budgets).
- **A tight topology spread.** The platform's default spread is loose (`maxSkew: 3`, schedule anyway), so all your replicas can still land on one node. Set `maxSkew: 1` across hostnames so a single drain removes at most one. See [Spread pods with topology constraints](/docs/hetzner/apalla/workloads/placement/topology-spread-constraints).

> [!NOTE]
> During a drain the platform respects the PodDisruptionBudget, but only up to `nodeDrainTimeoutSeconds`, which defaults to 180 seconds per pool. Once the timeout elapses the drain proceeds and the remaining pods go with the node. The budget provides an orderly move, not an indefinite hold, which is why the spread matters too.

If your application needs longer to finish in-flight requests, raise the timeout for its pool. That is a cluster-level change, set under that pool's `deletion` block in `spec.topology.workers.machineDeployments`, not in the Deployment. It applies to that pool alone; every other keeps the 180-second default.

```yaml title="cluster.yaml (excerpt)"
spec:
  topology:
    workers:
      machineDeployments:
        - class: workeramd64hcloud
          name: md-0
          replicas: 3
          deletion:
            nodeDrainTimeoutSeconds: 600
```

## Every setting in one manifest

A Deployment carrying all five, plus its PodDisruptionBudget. The example uses `nginx:stable`, which serves `/` on port 80, so the probes pass as written; swap the image and point the probes at your own health endpoint.

```yaml title="production-workload.yaml"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: team-web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: app
          image: nginx:stable
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 10
          resources:
            requests: {cpu: "250m", memory: "256Mi"}
            limits: {memory: "256Mi"}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: team-web
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web
```

## One drain, narrated

The five settings read like a checklist, but they are really a relay. Watch them hand off during one node replacement. You start a Kubernetes upgrade, the rollout reaches `worker-2`, and one of your three pods lives there.

```mermaid
sequenceDiagram
    participant Roll as Rolling upgrade
    participant New as Replacement node
    participant PDB as PodDisruptionBudget
    participant Sched as Scheduler
    participant LB as Service endpoints
    Roll->>New: bring up a fresh node first (maxSurge 1, maxUnavailable 0)
    New-->>Roll: node Ready and empty
    Roll->>PDB: cordon worker-2, ask to evict web-b
    PDB-->>Roll: one eviction allowed (2 of 3 stay up)
    Roll->>LB: web-b marked not-ready, drops from rotation
    Roll->>Sched: web-b terminates, replacement pod needed
    Sched->>New: requests + maxSkew 1 pick the empty new node
    New->>LB: new pod passes readiness, joins rotation
```

A cloud pool upgrades by surging: it brings the replacement node up and waits for it to be Ready _before_ it touches `worker-2` (the pool rolls with `maxSurge: 1`, `maxUnavailable: 0`), so an empty node is always waiting when the drain starts. The **PDB** refuses to let a second pod go while `web-b` is still leaving, so you never drop below two. The **readiness probe** pulls `web-b` from the Service before it dies, so no request lands on a terminating pod. The **replica count** keeps the other two serving throughout. The **resource requests** and **`maxSkew: 1`** then steer the replacement onto that fresh node, which has the most room and is not already running `web`, so the next drain is just as survivable. The handoff fits inside the 180-second window, and traffic never notices. Drop any setting and a link in the chain breaks.

## Verify

Do not take the diagram on faith. Drain a node and watch the relay run.

At rest, the pods should sit on three nodes with budget headroom:

```console
$ kubectl get pods -n team-web -l app=web -o wide
$ kubectl get pdb -n team-web web
```

On the PDB, `ALLOWED DISRUPTIONS 1` means a drain may evict one pod at a time, the healthy resting state. `0` means a pod is not ready and a drain will wait for it up to the 180-second timeout.

Now force the event yourself. Cordon and drain the node running one of the pods:

```console
$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
```

With `kubectl get pods -n team-web -o wide -w` open in another terminal, you should see exactly one pod leave, two stay `Running` throughout, and the replacement come up on a node that was not already running `web`. A replacement stuck `Pending` means the spread found no free node. Add a worker or relax the constraint to `ScheduleAnyway`. Uncordon the node when you are done.

## What this page does not cover

This is availability. Two things are separate:

- **Namespace guardrails.** Set a [LimitRange and ResourceQuota](/docs/hetzner/apalla/workloads/production/namespace-limitrange-and-resourcequota) once per namespace so a missing request cannot hurt neighbours.
- **Hardening.** [Set security contexts](/docs/hetzner/apalla/workloads/secure/set-security-contexts) to drop privileges, and [segment with network policies](/docs/hetzner/apalla/security/segment-with-network-policies) to limit what can reach your pods.