Survive node drains

Syself Autopilot drains and replaces nodes often, on every rolling upgrade and whenever self-healing replaces an unhealthy node. A bare Deployment loses traffic on each of those. This page lists every setting that keeps it serving. Apply all of them for anything serving real traffic; each has its own page for the detail. For how these rules behave during a drain and the graceful shutdown a stateless service adds, see Keep stateless workloads available .

The baseline 

Make sure all your production workloads pass the following checklist:

• At least two replicas. A single replica goes fully down the moment its node drains. A second keeps serving while the first moves.
• A readiness probe (and usually a liveness probe), so traffic only reaches a pod that can serve, and a stuck process gets restarted. See Configure health probes .
• Resource requests on every container. A request-less pod looks free to the scheduler and the autoscaler, so it gets packed onto a full node and evicted under pressure. See Set resource requests and limits .
• A PodDisruptionBudget, so a drain waits rather than take your last pod down. See Add a PodDisruptionBudget .
• A tight topology spread. The platform's default spread is loose (maxSkew: 3, schedule anyway), so all your replicas can still land on one node. Set maxSkew: 1 across hostnames so a single drain removes at most one. See Spread pods with topology constraints .

The drain timeout overrides the budget 

During a drain the platform respects the PodDisruptionBudget, but only up to nodeDrainTimeoutSeconds, which defaults to 180 seconds per pool. Once the timeout elapses the drain proceeds and the remaining pods go with the node. The budget provides an orderly move, not an indefinite hold, which is why the spread matters too.

If your application needs longer to finish in-flight requests, raise the timeout for its pool. That is a cluster-level change, set under that pool's deletion block in spec.topology.workers.machineDeployments, not in the Deployment. It applies to that pool alone; every other keeps the 180-second default.

				spec:
  topology:
    workers:
      machineDeployments:
        - class: workeramd64hcloud
          name: md-0
          replicas: 3
          deletion:
            nodeDrainTimeoutSeconds: 600
			

Every setting in one manifest 

A Deployment carrying all five, plus its PodDisruptionBudget. The example uses nginx:stable, which serves / on port 80, so the probes pass as written; swap the image and point the probes at your own health endpoint.

				apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: team-web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: app
          image: nginx:stable
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 10
          resources:
            requests: {cpu: "250m", memory: "256Mi"}
            limits: {memory: "256Mi"}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: team-web
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web
			

One drain, narrated 

Each baseline setting mentioned above handles a different stage of a single node replacement. The diagram below traces one such replacement, stage by stage.

You start a Kubernetes upgrade. It replaces nodes one at a time, and when it reaches worker-2, one of your three replicas is running there and has to move to another node.

sequenceDiagram
    participant Roll as Rolling upgrade
    participant New as Replacement node
    participant PDB as PodDisruptionBudget
    participant Sched as Scheduler
    participant LB as Service endpoints
    Roll->>New: bring up a fresh node first (maxSurge 1, maxUnavailable 0)
    New-->>Roll: node Ready and empty
    Roll->>PDB: cordon worker-2, ask to evict web-b
    PDB-->>Roll: one eviction allowed (2 of 3 stay up)
    Roll->>LB: web-b marked not-ready, drops from rotation
    Roll->>Sched: web-b terminates, replacement pod needed
    Sched->>New: requests + maxSkew 1 pick the empty new node
    New->>LB: new pod passes readiness, joins rotation

A cloud pool upgrades by surging: it brings the replacement node up and waits for it to be Ready before it touches worker-2 (the pool rolls with maxSurge: 1, maxUnavailable: 0), so an empty node is always waiting when the drain starts. The PDB refuses to let a second pod go while web-b is still leaving, so you never drop below two. The readiness probe pulls web-b from the Service before it dies, so no request lands on a terminating pod. The replica count keeps the other two serving throughout. The resource requests and maxSkew: 1 then steer the replacement onto that fresh node, which has the most room and is not already running web, so the next drain is just as survivable. The handoff fits inside the 180-second window, and traffic is not interrupted.

Verify 

Drain a node and watch the sequence run.

At rest, the pods should sit on three nodes with budget headroom:

				$ kubectl get pods -n team-web -l app=web -o wide
$ kubectl get pdb -n team-web web
			

On the PDB, ALLOWED DISRUPTIONS 1 means a drain may evict one pod at a time, the healthy resting state. 0 means a pod is not ready and a drain will wait for it up to the 180-second timeout.

On Autopilot, a drain is triggered by deleting a node's Machine object: Cluster API cordons and drains the node for you, with no extra steps inside the workload cluster. To reproduce that eviction yourself without replacing a node, drain one of the nodes running a pod manually:

				$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
			

With kubectl get pods -n team-web -o wide -w open in another terminal, you should see exactly one pod leave, two stay Running throughout, and the replacement come up on a node that was not already running web. A replacement stuck Pending means the spread found no free node. Add a worker or relax the constraint to ScheduleAnyway. Uncordon the node when you are done.

What this page does not cover 

The five settings keep enough replicas serving through a drain. Four things build on that, each with its own page:

• Graceful shutdown. The five keep the application up as a whole; they do not make each terminating pod exit cleanly. Add a preStop hook and SIGTERM handling so in-flight requests finish instead of dropping mid-flight. See Handle graceful shutdown .
• Stateful data. Everything above keeps stateless pods serving; none of it protects the data a database or a volume holds. Replicas and a PodDisruptionBudget move a pod, not the bytes under it. If your workload has state, first keep the data across a node move with stateful placement , then back it up and rehearse the restore, because a backup you have never restored is not yet a backup. See Back up your workloads and Run a restore drill .
• Namespace guardrails. Set a LimitRange and ResourceQuota once per namespace so a missing request cannot hurt neighbours.
• Hardening. Set security contexts to drop privileges, and segment with network policies to limit what can reach your pods.

Where to go next 

If you follow all the precautions mentioned above, your stateless workloads will be protected from node drains. Where you head from here depends on what you are building:

• Protect your other workload types. An ingress controller and a database need the same care, each with its own requirements. See Keep ingress and load balancers available and Keep databases available .
• Catch it before an upgrade does. A drain is exactly what an upgrade puts your workload through, so confirm your budgets hold first. See Add a PodDisruptionBudget .
• Handle traffic, not just drains. Add a Horizontal Pod Autoscaler , but keep its minReplicas at two, so at a quiet time you don't scale down to one and the PodDisruptionBudget has nothing left to protect. Turn on the cluster autoscaler so a drained pod always has a node to land on instead of sitting Pending.
• Know when something breaks. This page creates the very signals worth alerting on: a PodDisruptionBudget stuck at 0, an OOMKilled container, a restart loop. Wire them up with platform alert rules and kube-state metrics .
• Ship it the same way every time. Put the manifest under GitOps with Argo CD so configuration drift never quietly reopens a gap you just closed.