Run GPU workloads
Running a GPU workload on Syself Autopilot takes no driver setup. When a bare-metal server with a supported NVIDIA GPU joins the cluster, the driver, the CDI spec, and the device plugin are already in place, because they ship inside the sealed node OS . You request a GPU the way you request memory, and the pod runs. There is no operator to install, no driver DaemonSet to reconcile, and no image variant to pick.
What happens when a GPU server joins
A GPU server comes online ready to use. The driver, CDI spec, and device plugin ship inside the node OS, so a node that reaches Ready can run CUDA right away. Skip ahead to Schedule a GPU pod to run one.
Supported cards
A server gets GPU support only for cards the platform recognizes. See supported GPU cards for the current list. Any other NVIDIA card is treated as a plain display adapter and receives no GPU support: no GPU driver, no device plugin, no advertised resource.
Schedule a GPU pod
Request nvidia.com/gpu in the container's limits. The scheduler places the pod on a GPU node.
apiVersion: v1
kind: Pod
metadata:
name: gpu-check
spec:
restartPolicy: Never
containers:
- name: cuda
image: nvidia/cuda:12.4.1-base-ubuntu22.04
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1 # whole GPUs only; this request is what pins the pod to a GPU node
You do not need a nodeSelector. The nvidia.com/gpu request already steers the pod, because non-GPU nodes never advertise the resource and so never match.
Run the pod, then do kubectl logs gpu-check. If nvidia-smi lists the card model and driver version from inside the container, the GPU is fully wired through to the pod, not just allocated to it.
Read Capacity against Allocated
Every GPU advertises a fixed slice count:
$ kubectl describe node <gpu-node> | grep nvidia.com/gpu
nvidia.com/gpu: 4 # Capacity and Allocatable: the fixed slice count
Capacity and Allocatable report the total slices (commonly 4), and they do not shrink as pods claim them. To see what is actually in use, read the Allocated resources section further down the same describe output. Allocatable: 0 means the device plugin is missing or unhealthy, not that the card is busy. What a slice is, and how many pods should share one card, are covered in Share a GPU across pods .
Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
Pod stays Pending, no GPU nodes found | No worker has a detected GPU | kubectl get nodes -L autopilot.syself.com/gpu; check the server has a supported card |
Pod stays Pending | No free GPU slice on any node | kubectl describe node <gpu-node> and check Allocated resources |
| Pod fails to start, GPU absent in the container | gpu-activate did not complete | Check the node condition GpuActivationFailed. See Node health conditions |
| GPU errors at runtime | Hardware fault | Check the node condition GpuFallenOffBus |
Related: Add GPU nodes for the hardware and a dedicated pool, GPU not scheduling to fix a pod stuck Pending, and Share a GPU across pods for slicing. For adding GPU servers, see Add bare-metal servers .
Control retries and clean up finished jobs
Finished Jobs quietly bloat etcd and slow the API server. Set a TTL so they clean themselves up, and bound retries so a failing Job cannot loop forever.
Share a GPU across pods
A slice is a scheduling token, not a fraction of the card. Learn when co-scheduling is safe and how to keep a whole GPU to one workload.