This page covers the decisions specific to running Jobs on Syself Autopilot: surviving node drains, choosing a runtime for untrusted code, placing a Job on the right node type, and scaling on demand. For the Job API itself (`completions`, `parallelism`, `backoffLimit`, `restartPolicy`, and reading Job status), see the [Kubernetes Jobs documentation](https://kubernetes.io/docs/concepts/workloads/controllers/job/).

## Surviving node drains

When a drain evicts the Job's pod, a fresh pod starts elsewhere and the work restarts from the beginning. Either the work is safe to repeat as often as needed, because it does nothing critical, or you have to make it idempotent by keeping state outside the pod that a new run can read to pick up where the last one left off. Either way, treat an interrupted run as normal, not as a failure.

Eviction is not instant. Before the pod is killed, it gets a grace window to shut down cleanly. That window is `nodeDrainTimeoutSeconds`, which defaults to 180 seconds per worker pool. If your Job needs longer to finish its work and exit, [raise `nodeDrainTimeoutSeconds`](/docs/hetzner/apalla/workloads/production/run-a-production-ready-workload) on the pool it runs on, and have the Job [shut down gracefully](/docs/hetzner/apalla/workloads/production/graceful-shutdown) within that window.

## Is my workload safe to run?

A Job that runs untrusted or risky code should not share a node's kernel with everything else on it. Run it in the secure runtime where each pod gets its own lightweight VM, so a container or kernel escape stays trapped inside that VM instead of reaching the host or its neighbours. You opt in per workload with `runtimeClassName: secure`.

For a full explanation of each runtime, when to reach for it, and how they compare, see [Workload runtimes and isolation](/docs/hetzner/apalla/security/workload-runtimes-and-isolation).

## Influence where a Job is scheduled

Most Jobs can run on any worker node, and the scheduler places them for you. However, some have specific hardware or performance requirements that the default placement does not satisfy. A Job that needs a GPU, or one that drives a database or other heavy work, has to be placed on a node that meets those needs.

- **GPU work** (training, inference, batch transcode) needs a GPU node. Request the GPU in the pod spec and the scheduler routes it to a GPU pool. See [Run GPU workloads](/docs/hetzner/apalla/workloads/specialized/run-gpu-workloads); when several Jobs can safely share one card, see [Share a GPU](/docs/hetzner/apalla/workloads/specialized/share-a-gpu).
- **Heavy or stateful work** (a large import, a memory-intensive rollup) usually runs best on bare metal for steady performance, while short bursty Jobs fit cloud nodes. See [Mix cloud and bare metal](/docs/hetzner/apalla/servers-and-nodes/pools/mix-cloud-and-bare-metal).

To pin a Job to a node type in general, use a node selector or affinity as covered in [Steer workload placement](/docs/hetzner/apalla/workloads/placement/steer-workload-placement).

## Scale Jobs on demand

When a burst of Jobs asks for more CPU or memory than the cluster currently has, their pods sit `Pending`. With [Cluster Autoscaler](/docs/hetzner/apalla/clusters/configure/autoscale-a-cluster) enabled, that pending state provisions new Hetzner Cloud nodes on demand, and once the Jobs finish and the nodes fall idle, the autoscaler removes them again, so you pay for the extra capacity only while the work runs.

This helps most with batch work that arrives in bursts. Consider a queue of data-export Jobs that is nearly empty most of the day, until many users trigger exports at once and dozens of Jobs are created together. Their pods go `Pending` for lack of capacity, the autoscaler adds cloud nodes to run them, and those nodes are removed once the queue drains. Keep each Job's resource requests accurate, since the autoscaler sizes the new nodes from what the pending pods ask for.

> [!NOTE]
> Only cloud pools scale this way. Bare-metal nodes, including GPU nodes, are fixed capacity that you provision ahead of time, so size those pools for your peak load.

## Related

- [Control retries and clean up finished jobs](/docs/hetzner/apalla/workloads/batch/backoff-and-ttl-cleanup): cap retries with `backoffLimit` and `activeDeadlineSeconds`, and delete finished Jobs automatically.