When Syself Autopilot replaces a node, which is routine work rather than an incident, every pod on it is asked to stop. If your application exits the instant it is told to, it drops whatever requests were in flight. Graceful shutdown is how you finish those requests first, so a node replacement is invisible to your users instead of a burst of 502s.

## Two lanes run in parallel

Deleting a pod kicks off two things at once, in two different parts of the cluster. The pod's process gets a chance to wind down, and the Service quietly stops sending it traffic. They run in parallel, and that parallelism is the key point here:

```mermaid
flowchart LR
  Del["Pod deleted"]
  Del --> P1
  Del --> S1
  subgraph process["Process lane: yours to handle (grace period covers all of it)"]
    direction LR
    P1["preStop:<br/>sleep 5s"] --> P2["SIGTERM<br/>drain, stop accepting"] --> P4["SIGKILL<br/>when grace period ends"]
  end
  subgraph endpoints["Endpoints lane: cluster handles"]
    direction LR
    S1["Removed from<br/>Service endpoints"] --> S2["Change propagates to<br/>every kube-proxy / LB"] --> S3["No new traffic<br/>reaches the pod"]
  end
  P1 -. "the sleep holds SIGTERM<br/>until this lands" .-> S3
```

The process lane is yours to handle. On SIGTERM the application should stop accepting new work, finish what it already has open, and exit before the grace period runs out. A process that ignores SIGTERM is killed hard at the end. But the endpoints lane is where the subtle bug lives.

## The endpoint deregistration race

Notice that nothing synchronises the two lanes. Endpoint removal and SIGTERM both start the moment the pod is deleted, and endpoint removal is not instant, because it has to propagate out to every kube-proxy and load balancer before they stop picking the pod. So there is a window where your process has already received SIGTERM and started shutting down, yet a proxy that has not caught up is still routing fresh requests to it. That is the race, and it is the real reason `preStop` sleeps exist.

Two moves close it:

- **Fail readiness first.** As soon as the application begins shutting down, its readiness probe should fail, which is the signal that pulls the pod out of the endpoint set. See [Configure health probes](/docs/hetzner/apalla/workloads/production/health-probes).
- **Hold the process open with a short `preStop` sleep** so SIGTERM is delayed long enough for the endpoint removal to actually land everywhere. Five seconds is plenty for most clusters.

## The settings that fix it

```yaml
spec:
  terminationGracePeriodSeconds: 45 # total budget to SIGKILL; must cover the preStop sleep plus your slowest request
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"] # delay SIGTERM so endpoint removal lands first
```

`terminationGracePeriodSeconds` (default 30) is the total budget from the moment the pod is deleted until SIGKILL. The `preStop` sleep runs _first_ and provides the delay the race above needs, but it runs _inside_ that budget rather than on top of it: a 5-second sleep leaves 40 seconds of a 45-second budget for SIGTERM handling. Set the budget longer than the sleep plus your longest normal request, so an in-flight one can still finish. And in the application itself, handle SIGTERM: drain, then exit. Two of these are Kubernetes config; the third is code, and skipping it makes the other two pointless.

## Long-lived connections

A few seconds of grace is fine for HTTP request/response, but websockets, gRPC streams, and database sessions stay open by design. For those, either raise `terminationGracePeriodSeconds` enough for an orderly close, or have the application send a close frame or gRPC `GOAWAY` on SIGTERM so clients reconnect to a healthy pod on their own. Cutting them at SIGKILL is the one outcome to avoid.

## Verify

The only honest test is to reproduce a real drain. Point a steady stream of requests at the Service, then take a node out from under it manually:

```console
$ kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
```

Watch the client through the drain. Zero failed requests means shutdown is graceful. A burst of connection-resets or 502s means the application is exiting before it drains. Go back to the `preStop` hook and the SIGTERM handler. This is the same drain a [PodDisruptionBudget](/docs/hetzner/apalla/workloads/production/pod-disruption-budgets) paces and one piece of the full [production baseline](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload).