Configure health probes
Probes tell Kubernetes whether a pod can take traffic and whether it is still working. On a platform that drains and replaces nodes on its own schedule, that matters: a pod with no readiness probe takes requests the instant it exists, so every rollout and every node replacement drops a handful of requests. That is why probes sit in the production baseline , not the nice-to-have pile.
Most probe problems come from probes that are set up wrong, not from missing probes. These four cause the most trouble.
The mistakes to avoid#
No readiness probe at all. The most common and the most damaging. Without it the Service routes to the pod from the first millisecond, so requests fail while new pods warm up, a burst of 502s that coincides exactly with your deployments.
Liveness aimed at a dependency. This is the one that turns a minor problem into an incident.
Warning
Point liveness at your database and a single database blip restarts every pod at once. The kubelet kills each container, they all come back cold, they all hammer the recovering database together, and a small outage becomes a big one. Liveness must test only the process itself. A dependency check belongs in readiness, where failing just parks the pod out of rotation instead of destroying it.
Liveness too aggressive. A short periodSeconds and a low failureThreshold will restart a merely-slow application mid-work, and a genuinely slow starter loops: killed before it finishes booting, killed again, forever. Give it a startupProbe and a threshold that reflects how long it really takes.
A probe endpoint that lies. A /healthz hard-wired to return 200 passes every check and tells you nothing. The endpoint must reflect whether the application can actually serve, but stay cheap, or the check becomes its own source of load.
A shape that avoids all four:
readinessProbe:
httpGet:
path: /healthz # reflects real serving state, stays cheap
port: 8080
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /livez # process only, never a dependency
port: 8080
periodSeconds: 10
failureThreshold: 3
startupProbe:
httpGet:
path: /livez
port: 8080
periodSeconds: 5
failureThreshold: 30 # up to ~150s to start before liveness begins
Liveness and readiness checks do not begin until the startup probe first succeeds, so a slow boot cannot trip them.
Readiness on shutdown#
Readiness earns its keep on the way out, too. When a pod is terminating, readiness needs to fail before the process exits, so the Service pulls it from the endpoints and stops routing to it. Skip that handoff and in-flight requests get cut. The full sequence is in Handle graceful shutdown .
Each probe can be one of four kinds:
httpGet: a 2xx or 3xx response passes. Prefer this for HTTP applications.tcpSocket: passes if the port accepts a connection; for a plain TCP service.exec: passes if the command exits 0; only when neither above fits.grpc: passes if the application answers a gRPC HealthCheckRequest on the given port; for gRPC services.
Timing fields are identical across all four:
periodSeconds: how often to check.timeoutSeconds: how long to wait before a check counts as a miss.failureThreshold: how many consecutive misses before the probe acts.initialDelaySeconds: a fixed wait before the first check, mostly superseded by astartupProbe.
Run a production-ready workload
The one page that wires every setting a real Deployment needs to survive a node drain, then narrates a single drain to show each one earning its place.
Set resource requests and limits
Why the right default on this platform is a memory limit equal to the request, no CPU limit, and Burstable QoS.