Probes tell Kubernetes whether a pod can take traffic and whether it is still working. On a platform that drains and replaces nodes on its own schedule, that matters: a pod with no readiness probe takes requests the instant it exists, so every rollout and every node replacement drops a handful of requests. That is why probes sit in the [production baseline](/docs/hetzner/apalla/workloads/production/survive-node-drains), not the nice-to-have pile.

Most probe problems come from probes that are set up wrong, not from missing probes. These three cause the most trouble.

## The mistakes to avoid

**No readiness probe at all.** The most common and the most damaging. Without it the Service routes to the pod from the first millisecond, so requests fail while new pods warm up, a burst of 502s that coincides exactly with your deployments.

**Liveness aimed at a dependency.** This is the one that turns a minor problem into an incident.

> [!WARNING]
> If liveness checks your database, one database blip restarts every pod at once. This turns a small outage into a big one. Liveness must test only the process itself. Put dependency checks in readiness, so a failure just stops traffic to the pod instead of restarting it.

**Liveness too aggressive.** A short `periodSeconds` and a low `failureThreshold` restart an app that is only slow, not broken. A slow-booting app is worse: liveness kills it before it finishes starting, then kills it again on every retry. Add a `startupProbe` and set the threshold to match how long the app really takes to start.

The following yaml snippet avoids the mistakes:

```yaml
readinessProbe:
  httpGet:
    path: /healthz # reflects real serving state, stays cheap
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /livez # process only, never a dependency
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 30 # up to ~150s to start before liveness begins
```

Liveness and readiness checks do not begin until the startup probe first succeeds, so a slow boot cannot trip them. That is why a `startupProbe` beats a large `initialDelaySeconds`: a fixed delay is a guess you pay on every restart, while the startup probe ends the moment the app is actually up and then hands off to liveness.

Size `failureThreshold` from a measured boot, not a guess. The startup budget is `periodSeconds × failureThreshold`, so `periodSeconds: 5` with `failureThreshold: 30` allows 150 seconds. Time a real boot first:

```console
$ kubectl logs <pod> --timestamps | head -1                                   # container start
$ kubectl logs <pod> --timestamps | grep -m1 -iE "listening|ready|started"    # first serving log
```

Take the gap between those two timestamps, add roughly half again for a cold node or a slow image pull, and divide by `periodSeconds`. A 90-second boot works out to `ceil(90 × 1.5 / 5) = 27`, so `failureThreshold: 30` is a safe round number. Too low and a genuinely slow start is killed mid-boot and loops forever; too high only delays your noticing a boot that will never finish.

## Readiness on shutdown

When a pod is terminating, readiness needs to fail _before_ the process exits, so the Service pulls it from the endpoints and stops routing to it. Skip that handoff and in-flight requests get cut. The full sequence is in [Handle graceful shutdown](/docs/hetzner/apalla/workloads/production/graceful-shutdown).

## Verify

Verify a probe’s behavior instead of assuming it from the YAML. Apply the Deployment, then read the pod's events:

```console
$ kubectl describe pod <pod> | sed -n '/Events:/,$p'
```

A healthy start shows the startup probe passing and then the pod turning Ready. To see liveness act, make `/livez` fail and watch the restart counter climb:

```console
$ kubectl get pod <pod> -w
NAME   READY   STATUS    RESTARTS   AGE
web    1/1     Running   0          2m
web    0/1     Running   1          2m30s   # liveness fired, container restarted
```

`RESTARTS` climbing on a pod that should be healthy means liveness is too aggressive or aimed at a dependency. A pod stuck `0/1` that never restarts means readiness is failing while liveness passes, usually a dependency the readiness check needs. Both send you back to the three mistakes above.

<Collapsible title="Reference: probe types and timing knobs">

Each probe can be one of four kinds:

- `httpGet`: a 2xx or 3xx response passes. Prefer this for HTTP applications, because a returned status code proves the request path actually ran, where an open TCP port only proves the process is listening.
- `tcpSocket`: passes if the port accepts a connection; for a plain TCP service.
- `exec`: passes if the command exits 0; only when neither above fits.
- `grpc`: passes if the application answers a gRPC HealthCheckRequest on the given port; for gRPC services.

Timing fields are identical across all four:

- `periodSeconds`: how often to check.
- `timeoutSeconds`: how long to wait before a check counts as a miss.
- `failureThreshold`: how many consecutive misses before the probe acts.
- `initialDelaySeconds`: a fixed wait before the first check, mostly superseded by a `startupProbe`.

</Collapsible>