A control plane rarely fails outright because of finished Jobs. It becomes slow. A CronJob firing every minute leaves a Job and a pod object behind on every run, and nothing deletes them by default; over a week that is thousands of dead objects in etcd, the control-plane datastore. The API server reads them, backups grow, and list calls become heavier, until the cluster carries a lag no one can trace to a single cause. Leftover finished Jobs are among the most common reasons a control plane slowly degrades.

The fix is simple and permanent: give every Job a lifespan, and prevent failing Jobs from running longer than they should.

## Diagnose a bloated control plane from leftover Jobs

If a cluster feels sluggish and you suspect leftover Jobs, work through the causes from most to least likely.

<Steps>

1. Count finished Jobs across all namespaces. A handful is normal; hundreds or thousands is the symptom.
2. Find the source. It is almost always a CronJob on a frequent schedule with no cleanup.
3. Check whether those Jobs carry a TTL. Inspect a leftover Job's spec for `ttlSecondsAfterFinished`; if it is absent, nothing will ever remove it.
4. Set the TTL (below) on the offending `jobTemplate`, then delete the existing backlog so etcd sheds the accumulated objects.

</Steps>

## Clean up with ttlSecondsAfterFinished

Nothing removes a finished Job on its own. Set `ttlSecondsAfterFinished` and Kubernetes deletes the Job, pods included, that many seconds after it finishes:

```yaml
spec:
  ttlSecondsAfterFinished: 3600 # delete one hour after finishing
```

Treat this as mandatory on every Job, not as an optimization. One hour is enough to read logs after a failure; a routine success can use far less.

<Collapsible title="CronJob history limits versus TTL">

CronJobs add a second, overlapping control. `successfulJobsHistoryLimit` and `failedJobsHistoryLimit` cap how many finished Jobs are kept (defaults 3 and 1); a TTL on the `jobTemplate` deletes each after a set time. Rely on the low history defaults to bound the count; never raise them on a frequent schedule.

</Collapsible>

## Bound retries with backoffLimit

`backoffLimit` caps how many times a Job's pod may fail before the whole Job is marked failed:

```yaml
spec:
  backoffLimit: 4
```

Between attempts Kubernetes waits with exponential backoff (10s, 20s, 40s, and so on, capped at six minutes), so a fast-failing Job does not spin in a tight loop and overload the API server. Keep the limit low for work that cannot succeed on retry, such as a bad input or a missing dependency; a high limit only delays the inevitable.

## Skip retries that cannot succeed with podFailurePolicy

Some failures warrant a retry (a transient network error); some do not (the program exited with a "bad config" code). `podFailurePolicy` branches on the exit code so the Job fails fast instead of exhausting `backoffLimit`:

```yaml
spec:
  backoffLimit: 4
  podFailurePolicy:
    rules:
      - action: FailJob # do not retry
        onExitCodes:
          operator: In
          values: [42] # the app's "unrecoverable" code
```

An `onExitCodes` rule requires the pod template's `restartPolicy: Never` (the Job examples here use it); Kubernetes rejects `podFailurePolicy` on an `OnFailure` Job.

## Draining a stuck Job

A Job stuck retrying, or one whose pod refuses to terminate, can hold up a node drain and stall a cluster upgrade. Delete it directly with `kubectl delete job <name>` to release the pod, then fix the underlying failure before rerunning. A `backoffLimit` paired with `activeDeadlineSeconds` keeps most Jobs out of that state; set both before you need them, not while an upgrade is blocked.