This page covers the decisions specific to running Jobs on Syself Autopilot: surviving node drains, choosing a runtime for untrusted code, placing a Job on the right node type, and scaling on demand. For the Job API itself (completions, parallelism, backoffLimit, restartPolicy, and reading Job status), see the Kubernetes Jobs documentation⁠.

Surviving node drains 

When a drain evicts the Job's pod, a fresh pod starts elsewhere and the work restarts from the beginning. Either the work is safe to repeat as often as needed, because it does nothing critical, or you have to make it idempotent by keeping state outside the pod that a new run can read to pick up where the last one left off. Either way, treat an interrupted run as normal, not as a failure.

Eviction is not instant. Before the pod is killed, it gets a grace window to shut down cleanly. That window is nodeDrainTimeoutSeconds, which defaults to 180 seconds per worker pool. If your Job needs longer to finish its work and exit, raise nodeDrainTimeoutSeconds on the pool it runs on, and have the Job shut down gracefully within that window.

Is my workload safe to run? 

A Job that runs untrusted or risky code should not share a node's kernel with everything else on it. Run it in the secure runtime where each pod gets its own lightweight VM, so a container or kernel escape stays trapped inside that VM instead of reaching the host or its neighbours. You opt in per workload with runtimeClassName: secure.

For a full explanation of each runtime, when to reach for it, and how they compare, see Workload runtimes and isolation .

Influence where a Job is scheduled 

Most Jobs can run on any worker node, and the scheduler places them for you. However, some have specific hardware or performance requirements that the default placement does not satisfy. A Job that needs a GPU, or one that drives a database or other heavy work, has to be placed on a node that meets those needs.

• GPU work (training, inference, batch transcode) needs a GPU node. Request the GPU in the pod spec and the scheduler routes it to a GPU pool. See Run GPU workloads ; when several Jobs can safely share one card, see Share a GPU .
• Heavy or stateful work (a large import, a memory-intensive rollup) usually runs best on bare metal for steady performance, while short bursty Jobs fit cloud nodes. See Mix cloud and bare metal .

To pin a Job to a node type in general, use a node selector or affinity as covered in Steer workload placement .

Scale Jobs on demand 

When a burst of Jobs asks for more CPU or memory than the cluster currently has, their pods sit Pending. With Cluster Autoscaler enabled, that pending state provisions new Hetzner Cloud nodes on demand, and once the Jobs finish and the nodes fall idle, the autoscaler removes them again, so you pay for the extra capacity only while the work runs.

This helps most with batch work that arrives in bursts. Consider a queue of data-export Jobs that is nearly empty most of the day, until many users trigger exports at once and dozens of Jobs are created together. Their pods go Pending for lack of capacity, the autoscaler adds cloud nodes to run them, and those nodes are removed once the queue drains. Keep each Job's resource requests accurate, since the autoscaler sizes the new nodes from what the pending pods ask for.

Related 

• Control retries and clean up finished jobs : cap retries with backoffLimit and activeDeadlineSeconds, and delete finished Jobs automatically.