GPU not scheduling

A GPU pod stuck in Pending (the scheduler cannot place it on any node), or a card that never shows as schedulable, comes down to one of three causes: the node was never recognized as a GPU node, the driver did not activate, or the pod's request is wrong. Check them in that order.

If all three check out and the pod is still Pending, no GPU node has a free slice left. See Is the request well-formed?.

flowchart TD
  A["GPU pod Pending"] --> B{"Node labelled<br/>autopilot.syself.com/gpu=true?"}
  B -->|No| B1["Card not on allowlist,<br/>no device plugin"]
  B -->|Yes| C{"nvidia.com/gpu<br/>capacity greater than 0?"}
  C -->|No| C1["Activation problem,<br/>check gpu-activate"]
  C -->|Yes| D{"Request nvidia.com/gpu: 1<br/>in limits?"}
  D -->|No| D1["Fix the request"]
  D -->|Yes| E{"Free slice on<br/>any GPU node?"}
  E -->|No| E1["All slices taken,<br/>add a node or free one"]
  E -->|Yes| F["Pod schedules"]

Is the node labelled?#

Syself Autopilot labels a node autopilot.syself.com/gpu=true only when its card matches the supported allowlist by PCI vendor and device ID. See supported GPU cards for the current list.

The NVIDIA device plugin is a managed cluster component. It is the piece that tells Kubernetes a node has GPUs, so pods can request them, and it deploys only on labelled nodes. No label means no device plugin, and no schedulable GPU.

		$ kubectl get nodes -L autopilot.syself.com/gpu
	

If the label is missing, the card is not on the allowlist and is treated as a plain display adapter. GPUs are bare metal only; there is no Hetzner Cloud GPU type.

Did the driver activate?#

If the node is labelled but nvidia.com/gpu capacity is 0, or a pod still will not schedule, check the GPU conditions and the activation service. A node condition is a status field Kubernetes keeps on the node object.

GpuActivationFailed means the GPU is present but activation did not finish, so the container runtime cannot use it yet. It only raises a flag; nothing reboots or replaces the node for it. Read the activation logs:

		$ journalctl -u gpu-activate.service -b
	

GpuFallenOffBus is a hardware fault, usually from power, heat, or a failing card. Rescheduling and reboots do not fix it, and Syself does not replace the node for it automatically. If you see it, the card needs attention: contact Hetzner to have it checked or swapped.

See Node problem detection and conditions for how to read these.

Is the request well-formed?#

Request exactly one GPU per container, in limits:

		resources:
  limits:
    nvidia.com/gpu: 1
	

A request for nvidia.com/gpu: 2 is rejected, because the device plugin runs with failRequestsGreaterThanOne. One GPU per container is the rule. Check what the node advertises:

		$ kubectl describe node <gpu-node> | grep nvidia.com/gpu
	

Capacity and Allocatable show the fixed slice count per node (normally 4, see When a card is shared). They do not shrink as pods claim slices, so a nonzero Allocatable does not mean a slice is free.

If the label, activation, and request all check out and the pod is still Pending, every slice on every GPU node is already claimed. Read the Allocated resources section further down in the same describe output, or list what already runs on the node:

		$ kubectl describe node <gpu-node>
$ kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
	

Add another GPU node, or free a slice by removing a pod.

When a card is shared#

More than one pod can land on the same card, and co-scheduled pods share its memory with no quota between them. So if a GPU job dies with a CUDA out-of-memory error, check what else is running on the card before you suspect the job. For how slicing and shared memory work, see Share a GPU across pods .

To keep a card to a single pod, give the exclusive GPU servers their own pool and steer only that workload to it with a nodeSelector (Syself Autopilot does not support node taints). See Add GPU nodes and Node taints are not supported .

Collecting GPU logs#

		$ kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
$ journalctl -u gpu-activate.service -b
	

If a job runs slower than expected, or fails with a CUDA out-of-memory error, the kubectl get pods command above shows what else is scheduled on the card. A shared card is the usual cause.

From inside a GPU pod, nvidia-smi confirms the driver and card are visible.

Still stuck once the label, activation, and request all check out? Add GPU nodes covers the supported hardware, Run GPU workloads covers scheduling a pod, and Where to start troubleshooting covers everything else.