Run a production-ready workload

Syself Autopilot drains and replaces nodes often, on every rolling upgrade and whenever self-healing replaces an unhealthy node. A bare Deployment loses traffic on each of those. This is the one page that lists every setting that keeps it serving. Apply all of them for anything serving real traffic; each has its own page for the detail. For how these rules behave during a drain and the graceful shutdown a stateless service adds, see Keep stateless workloads available .

The baseline

These five settings are non-optional for anything serving real traffic. Miss one and the others cannot cover for it:

• At least two replicas. A single replica goes fully down the moment its node drains. A second keeps serving while the first moves.
• A readiness probe (and usually a liveness probe), so traffic only reaches a pod that can serve, and a stuck process gets restarted. See Configure health probes .
• Resource requests on every container. A request-less pod looks free to the scheduler and the autoscaler, so it gets packed onto a full node and evicted under pressure. See Set resource requests and limits .
• A PodDisruptionBudget, so a drain waits rather than take your last pod down. See Add a PodDisruptionBudget .
• A tight topology spread. The platform's default spread is loose (maxSkew: 3, schedule anyway), so all your replicas can still land on one node. Set maxSkew: 1 across hostnames so a single drain removes at most one. See Spread pods with topology constraints .

If your application needs longer to finish in-flight requests, raise the timeout for its pool. That is a cluster-level change, set under that pool's deletion block in spec.topology.workers.machineDeployments, not in the Deployment. It applies to that pool alone; every other keeps the 180-second default.

		spec:
  topology:
    workers:
      machineDeployments:
        - class: workeramd64hcloud
          name: md-0
          replicas: 3
          deletion:
            nodeDrainTimeoutSeconds: 600
	

Every setting in one manifest

A Deployment carrying all five, plus its PodDisruptionBudget. The example uses nginx:stable, which serves / on port 80, so the probes pass as written; swap the image and point the probes at your own health endpoint.

		apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: team-web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: app
          image: nginx:stable
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 10
          resources:
            requests: {cpu: "250m", memory: "256Mi"}
            limits: {memory: "256Mi"}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: team-web
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web
	

One drain, narrated

The five settings read like a checklist, but they are really a relay. Watch them hand off during one node replacement. You start a Kubernetes upgrade, the rollout reaches worker-2, and one of your three pods lives there.

sequenceDiagram
    participant Roll as Rolling upgrade
    participant New as Replacement node
    participant PDB as PodDisruptionBudget
    participant Sched as Scheduler
    participant LB as Service endpoints
    Roll->>New: bring up a fresh node first (maxSurge 1, maxUnavailable 0)
    New-->>Roll: node Ready and empty
    Roll->>PDB: cordon worker-2, ask to evict web-b
    PDB-->>Roll: one eviction allowed (2 of 3 stay up)
    Roll->>LB: web-b marked not-ready, drops from rotation
    Roll->>Sched: web-b terminates, replacement pod needed
    Sched->>New: requests + maxSkew 1 pick the empty new node
    New->>LB: new pod passes readiness, joins rotation

A cloud pool upgrades by surging: it brings the replacement node up and waits for it to be Ready before it touches worker-2 (the pool rolls with maxSurge: 1, maxUnavailable: 0), so an empty node is always waiting when the drain starts. The PDB refuses to let a second pod go while web-b is still leaving, so you never drop below two. The readiness probe pulls web-b from the Service before it dies, so no request lands on a terminating pod. The replica count keeps the other two serving throughout. The resource requests and maxSkew: 1 then steer the replacement onto that fresh node, which has the most room and is not already running web, so the next drain is just as survivable. The handoff fits inside the 180-second window, and traffic never notices. Drop any setting and a link in the chain breaks.

Verify

Do not take the diagram on faith. Drain a node and watch the relay run.

At rest, the pods should sit on three nodes with budget headroom:

		$ kubectl get pods -n team-web -l app=web -o wide
$ kubectl get pdb -n team-web web
	

On the PDB, ALLOWED DISRUPTIONS 1 means a drain may evict one pod at a time, the healthy resting state. 0 means a pod is not ready and a drain will wait for it up to the 180-second timeout.

Now force the event yourself. Cordon and drain the node running one of the pods:

		$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
	

With kubectl get pods -n team-web -o wide -w open in another terminal, you should see exactly one pod leave, two stay Running throughout, and the replacement come up on a node that was not already running web. A replacement stuck Pending means the spread found no free node. Add a worker or relax the constraint to ScheduleAnyway. Uncordon the node when you are done.

What this page does not cover

The five settings keep enough replicas serving through a drain. Four things build on that, each with its own page:

• Graceful shutdown. The five keep the application up as a whole; they do not make each terminating pod exit cleanly. Add a preStop hook and SIGTERM handling so in-flight requests finish instead of dropping mid-flight. See Handle graceful shutdown .
• Stateful data. Everything above keeps stateless pods serving; none of it protects the data a database or a volume holds. Replicas and a PodDisruptionBudget move a pod, not the bytes under it. If your workload has state, first keep the data across a node move with stateful placement , then back it up and rehearse the restore, because a backup you have never restored is not yet a backup. See Back up your workloads and Run a restore drill .
• Namespace guardrails. Set a LimitRange and ResourceQuota once per namespace so a missing request cannot hurt neighbours.
• Hardening. Set security contexts to drop privileges, and segment with network policies to limit what can reach your pods.

Where to go next

If you follow all the precautions mentioned above, your stateless workloads will be protected from node drains. Where you head from here depends on what you are building:

• Protect your other workload types. An ingress controller and a database need the same care, each with its own twist. See Keep ingress and load balancers available and Keep databases available .
• Catch it before an upgrade does. A drain is exactly what an upgrade puts your workload through, so confirm your budgets hold first. See Plan maintenance and run prechecks .
• Handle traffic, not just drains. Add a Horizontal Pod Autoscaler , but keep its minReplicas at two, so at a quiet time you don't scale down to one and the PodDisruptionBudget has nothing left to protect. Turn on the cluster autoscaler so a drained pod always has a node to land on instead of sitting Pending.
• Know when something breaks. This page creates the very signals worth alerting on: a PodDisruptionBudget stuck at 0, an OOMKilled container, a restart loop. Wire them up with platform alert rules and kube-state metrics .
• Ship it the same way every time. Put the manifest under GitOps with Argo CD so configuration drift never quietly reopens a gap you just closed.