Configure health probes

Probes tell Kubernetes whether a pod can take traffic and whether it is still working. On a platform that drains and replaces nodes on its own schedule, that matters: a pod with no readiness probe takes requests the instant it exists, so every rollout and every node replacement drops a handful of requests. That is why probes sit in the production baseline , not the nice-to-have pile.

Most probe problems come from probes that are set up wrong, not from missing probes. These three cause the most trouble.

The mistakes to avoid 

No readiness probe at all. The most common and the most damaging. Without it the Service routes to the pod from the first millisecond, so requests fail while new pods warm up, a burst of 502s that coincides exactly with your deployments.

Liveness aimed at a dependency. This is the one that turns a minor problem into an incident.

Liveness too aggressive. A short periodSeconds and a low failureThreshold restart an app that is only slow, not broken. A slow-booting app is worse: liveness kills it before it finishes starting, then kills it again on every retry. Add a startupProbe and set the threshold to match how long the app really takes to start.

The following yaml snippet avoids the mistakes:

				readinessProbe:
  httpGet:
    path: /healthz # reflects real serving state, stays cheap
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /livez # process only, never a dependency
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 30 # up to ~150s to start before liveness begins
			

Liveness and readiness checks do not begin until the startup probe first succeeds, so a slow boot cannot trip them. That is why a startupProbe beats a large initialDelaySeconds: a fixed delay is a guess you pay on every restart, while the startup probe ends the moment the app is actually up and then hands off to liveness.

Size failureThreshold from a measured boot, not a guess. The startup budget is periodSeconds × failureThreshold, so periodSeconds: 5 with failureThreshold: 30 allows 150 seconds. Time a real boot first:

				$ kubectl logs <pod> --timestamps | head -1                                   # container start
$ kubectl logs <pod> --timestamps | grep -m1 -iE "listening|ready|started"    # first serving log
			

Take the gap between those two timestamps, add roughly half again for a cold node or a slow image pull, and divide by periodSeconds. A 90-second boot works out to ceil(90 × 1.5 / 5) = 27, so failureThreshold: 30 is a safe round number. Too low and a genuinely slow start is killed mid-boot and loops forever; too high only delays your noticing a boot that will never finish.

Readiness on shutdown 

When a pod is terminating, readiness needs to fail before the process exits, so the Service pulls it from the endpoints and stops routing to it. Skip that handoff and in-flight requests get cut. The full sequence is in Handle graceful shutdown .

Verify 

Verify a probe’s behavior instead of assuming it from the YAML. Apply the Deployment, then read the pod's events:

				$ kubectl describe pod <pod> | sed -n '/Events:/,$p'
			

A healthy start shows the startup probe passing and then the pod turning Ready. To see liveness act, make /livez fail and watch the restart counter climb:

				$ kubectl get pod <pod> -w
NAME   READY   STATUS    RESTARTS   AGE
web    1/1     Running   0          2m
web    0/1     Running   1          2m30s   # liveness fired, container restarted
			

RESTARTS climbing on a pod that should be healthy means liveness is too aggressive or aimed at a dependency. A pod stuck 0/1 that never restarts means readiness is failing while liveness passes, usually a dependency the readiness check needs. Both send you back to the three mistakes above.