Run a Job
A Deployment is built to run indefinitely; a Job is built to finish. It starts a pod, performs one piece of work (a migration, a nightly rollup), reports success or failure, and stops. The complication is where it runs. Nodes on this platform are recycled: by a Kubernetes upgrade you roll out, by autoscaling, or by a repair. Any Job running longer than a few minutes can therefore be evicted mid-run. Design for that first, ahead of parallelism or retries.
Surviving node drains#
Kubernetes does not checkpoint a Job pod for you. When a drain evicts it, the Job schedules a fresh pod elsewhere and the work restarts from the beginning; nothing is resumed. It must tolerate that: make it idempotent (safe to run twice from scratch, with no double-counting), or checkpoint progress externally and resume from the last mark. Treat an interrupted run as normal, not as the failure case.
Eviction is not instant. The pod receives a grace window to exit before the hard kill: nodeDrainTimeoutSeconds, a per-pool default of 180 seconds. A Job that needs longer to flush a partial write can raise the timeout on its pool in the Cluster topology.
Warning
Trapping SIGTERM for a graceful shutdown does not provide unlimited time. If the wind-down runs past the pool's nodeDrainTimeoutSeconds, the drain hard-kills the pod mid-cleanup; 180 seconds is the ceiling until you raise it.
A basic Job#
apiVersion: batch/v1
kind: Job
metadata:
name: import
spec:
backoffLimit: 4 # retry the work up to 4 times before failing the Job
template:
spec:
restartPolicy: Never # fresh pod per attempt; keeps failed pods' logs
containers:
- name: import
image: your/importer:tag
command: ["./run-import"] # must be safe to re-run from scratch
To bound wall-clock time with activeDeadlineSeconds, or have finished Jobs delete themselves, see Control retries and clean up finished jobs .
Parallelism and completions#
Two fields scale one pod into many. completions is how many successful runs the Job needs; parallelism is how many run at once. Set completions: 10 and parallelism: 3, and ten work items run three at a time. When each pod needs to know which slice it owns, add completionMode: Indexed: every pod reads a JOB_COMPLETION_INDEX (0 to 9 here) and selects its own partition.
restartPolicy: Never or OnFailure#
A Job pod must use Never or OnFailure, never Always. The choice depends on logs. Never starts a fresh pod per attempt and leaves the failed ones in place, so their logs stay readable, which is usually what you want while debugging. OnFailure restarts the container in place: fewer objects, but the failed container's logs are lost.
Reading the result#
Applying a Job says nothing about whether it ran. Its COMPLETIONS column does: kubectl get job import reads 1/1 once the work has finished and 0/1 while it is still running, or still failing. Those two states look identical. The distinguishing signal is the pods. With restartPolicy: Never, each attempt is a fresh pod, so a failing Job leaves a trail: kubectl get pods -l job-name=import shows failed pods stacking up while completions stay at 0/1, whereas a slow Job shows a single pod still Running. Pull the logs from a failed pod for the reason, and check backoffLimit for how many attempts remain. (With OnFailure the signal is instead the restart count climbing on one pod, seen in the pod's RESTARTS column; kubectl get job has no such column.)