Configure health probes

Probes tell Kubernetes whether a pod can take traffic and whether it is still working. On a platform that drains and replaces nodes on its own schedule, that matters: a pod with no readiness probe takes requests the instant it exists, so every rollout and every node replacement drops a handful of requests. That is why probes sit in the production baseline , not the nice-to-have pile.

Most probe problems come from probes that are set up wrong, not from missing probes. These four cause the most trouble.

The mistakes to avoid

No readiness probe at all. The most common and the most damaging. Without it the Service routes to the pod from the first millisecond, so requests fail while new pods warm up, a burst of 502s that coincides exactly with your deployments.

Liveness aimed at a dependency. This is the one that turns a minor problem into an incident.

Liveness too aggressive. A short periodSeconds and a low failureThreshold will restart a merely-slow application mid-work, and a genuinely slow starter loops: killed before it finishes booting, killed again, forever. Give it a startupProbe and a threshold that reflects how long it really takes.

A probe endpoint that lies. A /healthz hard-wired to return 200 passes every check and tells you nothing. The endpoint must reflect whether the application can actually serve, but stay cheap, or the check becomes its own source of load.

A shape that avoids all four:

		readinessProbe:
  httpGet:
    path: /healthz # reflects real serving state, stays cheap
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /livez # process only, never a dependency
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 30 # up to ~150s to start before liveness begins
	

Liveness and readiness checks do not begin until the startup probe first succeeds, so a slow boot cannot trip them. That is why a startupProbe beats a large initialDelaySeconds: a fixed delay is a guess you pay on every restart, while the startup probe ends the moment the app is actually up and then hands off to liveness.

Size failureThreshold from a measured boot, not a guess. The startup budget is periodSeconds × failureThreshold, so periodSeconds: 5 with failureThreshold: 30 allows 150 seconds. Time a real boot first:

		$ kubectl logs <pod> --timestamps | head -1                                   # container start
$ kubectl logs <pod> --timestamps | grep -m1 -iE "listening|ready|started"    # first serving log
	

Take the gap between those two timestamps, add roughly half again for a cold node or a slow image pull, and divide by periodSeconds. A 90-second boot works out to ceil(90 × 1.5 / 5) = 27, so failureThreshold: 30 is a safe round number. Too low and a genuinely slow start is killed mid-boot and loops forever; too high only delays your noticing a boot that will never finish.

A readiness endpoint that tells the truth

A /healthz answers one question: can this pod serve a request right now? Check the things that would make a request fail, and nothing else.

• Honest and cheap. Confirm startup finished and any connection pool the app needs is live, then return 200. No downstream query on the hot path.
• Lies. return 200 unconditionally. It passes while the app is deadlocked, so the Service keeps routing into a pod that cannot serve, the exact failure the probe was meant to catch.
• Too expensive. A real database query on every probe. At periodSeconds: 5 across every replica that is constant load, and a slow dependency now fails readiness even when the app itself is fine.

Readiness on shutdown

Readiness earns its keep on the way out, too. When a pod is terminating, readiness needs to fail before the process exits, so the Service pulls it from the endpoints and stops routing to it. Skip that handoff and in-flight requests get cut. The full sequence is in Handle graceful shutdown .

Verify

Watch a probe do its job rather than trust the YAML. Apply the Deployment, then read the pod's events:

		$ kubectl describe pod <pod> | sed -n '/Events:/,$p'
	

A healthy start shows the startup probe passing and then the pod turning Ready. To see liveness act, make /livez fail and watch the restart counter climb:

		$ kubectl get pod <pod> -w
NAME   READY   STATUS    RESTARTS   AGE
web    1/1     Running   0          2m
web    0/1     Running   1          2m30s   # liveness fired, container restarted
	

RESTARTS climbing on a pod that should be healthy means liveness is too aggressive or aimed at a dependency. A pod stuck 0/1 that never restarts means readiness is failing while liveness passes, usually a dependency the readiness check needs. Both send you back to the four mistakes above.