Run a production-ready workload

Syself Autopilot drains and replaces nodes often, on every rolling upgrade and whenever self-healing replaces an unhealthy node. A bare Deployment loses traffic on each of those. This is the one page that lists every setting that keeps it serving. Apply all of them for anything serving real traffic; each has its own page for the detail. For how these rules behave during a drain and the graceful shutdown a stateless service adds, see Keep stateless workloads available .

The baseline#

Five settings, none optional for real traffic. Miss one and the others cannot cover for it.

• At least two replicas. A single replica goes fully down the moment its node drains. A second keeps serving while the first moves.
• A readiness probe (and usually a liveness probe), so traffic only reaches a pod that can serve, and a stuck process gets restarted. See Configure health probes .
• Resource requests on every container. A request-less pod looks free to the scheduler and the autoscaler, so it gets packed onto a full node and evicted under pressure. See Set resource requests and limits .
• A PodDisruptionBudget, so a drain waits rather than take your last pod down. See Add a PodDisruptionBudget .
• A tight topology spread. The platform's default spread is loose (maxSkew: 3, schedule anyway), so all your replicas can still land on one node. Set maxSkew: 1 across hostnames so a single drain removes at most one. See Spread pods with topology constraints .

If your application needs longer to finish in-flight requests, raise the timeout for its pool. That is a cluster-level change, set under that pool's deletion block in spec.topology.workers.machineDeployments, not in the Deployment. It applies to that pool alone; every other keeps the 180-second default.

		spec:
  topology:
    workers:
      machineDeployments:
        - class: workeramd64hcloud
          name: md-0
          replicas: 3
          deletion:
            nodeDrainTimeoutSeconds: 600
	

Every setting in one manifest#

A Deployment carrying all five, plus its PodDisruptionBudget. The example uses nginx:stable, which serves / on port 80, so the probes pass as written; swap the image and point the probes at your own health endpoint.

		apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: team-web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: app
          image: nginx:stable
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 5
          livenessProbe:
            httpGet: {path: /, port: 80}
            periodSeconds: 10
          resources:
            requests: {cpu: "250m", memory: "256Mi"}
            limits: {memory: "256Mi"}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
  namespace: team-web
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: web
	

One drain, narrated#

The five settings read like a checklist, but they are really a relay. Watch them hand off during one node replacement. You start a Kubernetes upgrade, the rollout reaches worker-2, and one of your three pods lives there.

sequenceDiagram
    participant Roll as Rolling upgrade
    participant New as Replacement node
    participant PDB as PodDisruptionBudget
    participant Sched as Scheduler
    participant LB as Service endpoints
    Roll->>New: bring up a fresh node first (maxSurge 1, maxUnavailable 0)
    New-->>Roll: node Ready and empty
    Roll->>PDB: cordon worker-2, ask to evict web-b
    PDB-->>Roll: one eviction allowed (2 of 3 stay up)
    Roll->>LB: web-b marked not-ready, drops from rotation
    Roll->>Sched: web-b terminates, replacement pod needed
    Sched->>New: requests + maxSkew 1 pick the empty new node
    New->>LB: new pod passes readiness, joins rotation

A cloud pool upgrades by surging: it brings the replacement node up and waits for it to be Ready before it touches worker-2 (the pool rolls with maxSurge: 1, maxUnavailable: 0), so an empty node is always waiting when the drain starts. The PDB refuses to let a second pod go while web-b is still leaving, so you never drop below two. The readiness probe pulls web-b from the Service before it dies, so no request lands on a terminating pod. The replica count keeps the other two serving throughout. The resource requests and maxSkew: 1 then steer the replacement onto that fresh node, which has the most room and is not already running web, so the next drain is just as survivable. The handoff fits inside the 180-second window, and traffic never notices. Drop any setting and a link in the chain breaks.

Verify#

Do not take the diagram on faith. Drain a node and watch the relay run.

At rest, the pods should sit on three nodes with budget headroom:

		$ kubectl get pods -n team-web -l app=web -o wide
$ kubectl get pdb -n team-web web
	

On the PDB, ALLOWED DISRUPTIONS 1 means a drain may evict one pod at a time, the healthy resting state. 0 means a pod is not ready and a drain will wait for it up to the 180-second timeout.

Now force the event yourself. Cordon and drain the node running one of the pods:

		$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
	

With kubectl get pods -n team-web -o wide -w open in another terminal, you should see exactly one pod leave, two stay Running throughout, and the replacement come up on a node that was not already running web. A replacement stuck Pending means the spread found no free node. Add a worker or relax the constraint to ScheduleAnyway. Uncordon the node when you are done.

What this page does not cover#

This is availability. Two things are separate:

• Namespace guardrails. Set a LimitRange and ResourceQuota once per namespace so a missing request cannot hurt neighbours.
• Hardening. Set security contexts to drop privileges, and segment with network policies to limit what can reach your pods.