Control retries and clean up finished jobs
A control plane rarely fails outright because of finished Jobs. It becomes slow. A CronJob firing every minute leaves a Job and a pod object behind on every run, and nothing deletes them by default; over a week that is thousands of dead objects in etcd, the control-plane datastore. The API server reads them, backups grow, and list calls become heavier, until the cluster carries a lag no one can trace to a single cause. Leftover finished Jobs are among the most common reasons a control plane slowly degrades.
The fix is simple and permanent: give every Job a lifespan, and prevent failing Jobs from running longer than they should.
Diagnose a bloated control plane from leftover Jobs
If a cluster feels sluggish and you suspect leftover Jobs, work through the causes from most to least likely.
- Count finished Jobs across all namespaces. A handful is normal; hundreds or thousands is the symptom.
- Find the source. It is almost always a CronJob on a frequent schedule with no cleanup.
- Check whether those Jobs carry a TTL. Inspect a leftover Job's spec for
ttlSecondsAfterFinished; if it is absent, nothing will ever remove it. - Set the TTL (below) on the offending
jobTemplate, then delete the existing backlog so etcd sheds the accumulated objects.
Clean up with ttlSecondsAfterFinished
Nothing removes a finished Job on its own. Set ttlSecondsAfterFinished and Kubernetes deletes the Job, pods included, that many seconds after it finishes:
spec:
ttlSecondsAfterFinished: 3600 # delete one hour after finishing
Treat this as mandatory on every Job, not as an optimization. One hour is enough to read logs after a failure; a routine success can use far less.
Bound retries with backoffLimit
backoffLimit caps how many times a Job's pod may fail before the whole Job is marked failed:
spec:
backoffLimit: 4
Between attempts Kubernetes waits with exponential backoff (10s, 20s, 40s, and so on, capped at six minutes), so a fast-failing Job does not spin in a tight loop and overload the API server. Keep the limit low for work that cannot succeed on retry, such as a bad input or a missing dependency; a high limit only delays the inevitable.
Skip retries that cannot succeed with podFailurePolicy
Some failures warrant a retry (a transient network error); some do not (the program exited with a "bad config" code). podFailurePolicy branches on the exit code so the Job fails fast instead of exhausting backoffLimit:
spec:
backoffLimit: 4
podFailurePolicy:
rules:
- action: FailJob # do not retry
onExitCodes:
operator: In
values: [42] # the app's "unrecoverable" code
An onExitCodes rule requires the pod template's restartPolicy: Never (the Job examples here use it); Kubernetes rejects podFailurePolicy on an OnFailure Job.
Draining a stuck Job
A Job stuck retrying, or one whose pod refuses to terminate, can hold up a node drain and stall a cluster upgrade. Delete it directly with kubectl delete job <name> to release the pod, then fix the underlying failure before rerunning. A backoffLimit paired with activeDeadlineSeconds keeps most Jobs out of that state; set both before you need them, not while an upgrade is blocked.