Probes tell Kubernetes whether a pod can take traffic and whether it is still working. On a platform that drains and replaces nodes on its own schedule, that matters: a pod with no readiness probe takes requests the instant it exists, so every rollout and every node replacement drops a handful of requests. That is why probes sit in the [production baseline](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload), not the nice-to-have pile. Most probe problems come from probes that are set up wrong, not from missing probes. These four cause the most trouble. ## The mistakes to avoid **No readiness probe at all.** The most common and the most damaging. Without it the Service routes to the pod from the first millisecond, so requests fail while new pods warm up, a burst of 502s that coincides exactly with your deployments. **Liveness aimed at a dependency.** This is the one that turns a minor problem into an incident. > [!WARNING] > Point liveness at your database and a single database blip restarts _every_ pod at once. The kubelet kills each container, they all come back cold, they all hammer the recovering database together, and a small outage becomes a big one. Liveness must test only the process itself. A dependency check belongs in readiness, where failing just parks the pod out of rotation instead of destroying it. **Liveness too aggressive.** A short `periodSeconds` and a low `failureThreshold` will restart a merely-slow application mid-work, and a genuinely slow starter loops: killed before it finishes booting, killed again, forever. Give it a `startupProbe` and a threshold that reflects how long it really takes. **A probe endpoint that lies.** A `/healthz` hard-wired to return 200 passes every check and tells you nothing. The endpoint must reflect whether the application can actually serve, but stay cheap, or the check becomes its own source of load. A shape that avoids all four: ```yaml readinessProbe: httpGet: path: /healthz # reflects real serving state, stays cheap port: 8080 periodSeconds: 5 failureThreshold: 3 livenessProbe: httpGet: path: /livez # process only, never a dependency port: 8080 periodSeconds: 10 failureThreshold: 3 startupProbe: httpGet: path: /livez port: 8080 periodSeconds: 5 failureThreshold: 30 # up to ~150s to start before liveness begins ``` Liveness and readiness checks do not begin until the startup probe first succeeds, so a slow boot cannot trip them. That is why a `startupProbe` beats a large `initialDelaySeconds`: a fixed delay is a guess you pay on every restart, while the startup probe ends the moment the app is actually up and then hands off to liveness. Size `failureThreshold` from a measured boot, not a guess. The startup budget is `periodSeconds × failureThreshold`, so `periodSeconds: 5` with `failureThreshold: 30` allows 150 seconds. Time a real boot first: ```console $ kubectl logs --timestamps | head -1 # container start $ kubectl logs --timestamps | grep -m1 -iE "listening|ready|started" # first serving log ``` Take the gap between those two timestamps, add roughly half again for a cold node or a slow image pull, and divide by `periodSeconds`. A 90-second boot works out to `ceil(90 × 1.5 / 5) = 27`, so `failureThreshold: 30` is a safe round number. Too low and a genuinely slow start is killed mid-boot and loops forever; too high only delays your noticing a boot that will never finish. ### A readiness endpoint that tells the truth A `/healthz` answers one question: can this pod serve a request right now? Check the things that would make a request fail, and nothing else. - **Honest and cheap.** Confirm startup finished and any connection pool the app needs is live, then return 200. No downstream query on the hot path. - **Lies.** `return 200` unconditionally. It passes while the app is deadlocked, so the Service keeps routing into a pod that cannot serve, the exact failure the probe was meant to catch. - **Too expensive.** A real database query on every probe. At `periodSeconds: 5` across every replica that is constant load, and a slow dependency now fails readiness even when the app itself is fine. ## Readiness on shutdown Readiness earns its keep on the way out, too. When a pod is terminating, readiness needs to fail _before_ the process exits, so the Service pulls it from the endpoints and stops routing to it. Skip that handoff and in-flight requests get cut. The full sequence is in [Handle graceful shutdown](/docs/hetzner/apalla/workloads/production/graceful-shutdown). ## Verify Watch a probe do its job rather than trust the YAML. Apply the Deployment, then read the pod's events: ```console $ kubectl describe pod | sed -n '/Events:/,$p' ``` A healthy start shows the startup probe passing and then the pod turning Ready. To see liveness act, make `/livez` fail and watch the restart counter climb: ```console $ kubectl get pod -w NAME READY STATUS RESTARTS AGE web 1/1 Running 0 2m web 0/1 Running 1 2m30s # liveness fired, container restarted ``` `RESTARTS` climbing on a pod that should be healthy means liveness is too aggressive or aimed at a dependency. A pod stuck `0/1` that never restarts means readiness is failing while liveness passes, usually a dependency the readiness check needs. Both send you back to the four mistakes above. Each probe can be one of four kinds: - `httpGet`: a 2xx or 3xx response passes. Prefer this for HTTP applications, because a returned status code proves the request path actually ran, where an open TCP port only proves the process is listening. - `tcpSocket`: passes if the port accepts a connection; for a plain TCP service. - `exec`: passes if the command exits 0; only when neither above fits. - `grpc`: passes if the application answers a gRPC HealthCheckRequest on the given port; for gRPC services. Timing fields are identical across all four: - `periodSeconds`: how often to check. - `timeoutSeconds`: how long to wait before a check counts as a miss. - `failureThreshold`: how many consecutive misses before the probe acts. - `initialDelaySeconds`: a fixed wait before the first check, mostly superseded by a `startupProbe`.