A GPU pod stuck in `Pending` (the scheduler cannot place it on any node), or a card that never shows as schedulable, comes down to one of three causes: the node was never recognized as a GPU node, the driver did not activate, or the pod's request is wrong. Check them in that order.

If all three check out and the pod is still `Pending`, no GPU node has a free slice left. See [Is the request well-formed?](#is-the-request-well-formed).

```mermaid
flowchart TD
  A["GPU pod Pending"]:::app --> B{"Node labelled<br/>autopilot.syself.com/gpu=true?"}:::decision
  B -->|No| B1["Card not on allowlist,<br/>no device plugin"]:::platform
  B -->|Yes| C{"nvidia.com/gpu<br/>capacity greater than 0?"}:::decision
  C -->|No| C1["Activation problem,<br/>check gpu-activate"]:::platform
  C -->|Yes| D{"Request nvidia.com/gpu: 1<br/>in limits?"}:::decision
  D -->|No| D1["Fix the request"]:::app
  D -->|Yes| E{"Free slice on<br/>any GPU node?"}:::decision
  E -->|No| E1["All slices taken,<br/>add a node or free one"]:::platform
  E -->|Yes| F["Pod schedules"]:::app
```

## Is the node labelled?

Syself Autopilot labels a node `autopilot.syself.com/gpu=true` only when its card matches the supported allowlist by PCI vendor and device ID. See [Add GPU nodes](/docs/hetzner/apalla/servers-and-nodes/provision/gpu-nodes) for the current list of supported cards.

The NVIDIA device plugin is a managed cluster component. It is the piece that tells Kubernetes a node has GPUs, so pods can request them, and it deploys **only** on labelled nodes. No label means no device plugin, and no schedulable GPU.

```console
$ kubectl get nodes -L autopilot.syself.com/gpu
```

If the label is missing, the card is not on the allowlist and is treated as a plain display adapter. GPUs are **bare metal only**; there is no Hetzner Cloud GPU type.

A card can load its driver without being on the allowlist, so `nvidia-smi` working on the host does not mean the node is labelled or schedulable.

## Did the driver activate?

If the node is labelled but `nvidia.com/gpu` capacity is `0`, or a pod still will not schedule, check the GPU conditions and the activation service. A node condition is a status field Kubernetes keeps on the node object.

**`GpuActivationFailed`** means the GPU is present but activation did not finish, so the container runtime cannot use it yet. It only raises a flag; nothing reboots or replaces the node for it. Read the activation logs:

```console
$ journalctl -u gpu-activate.service -b
```

**`GpuFallenOffBus`** is a hardware fault, usually from power, heat, or a failing card. Rescheduling and reboots do not fix it, and Syself does not replace the node for it automatically. If you see it, the card needs attention: contact Hetzner to have it checked or swapped.

See [Node problem detection and conditions](/docs/hetzner/apalla/servers-and-nodes/access/node-problem-detection-and-conditions) for how to read these.

## Is the request well-formed?

Request exactly one GPU per container, in `limits`:

```yaml
resources:
  limits:
    nvidia.com/gpu: 1
```

A request for `nvidia.com/gpu: 2` is **rejected**, because the device plugin runs with `failRequestsGreaterThanOne`. One GPU per container is the rule. Check what the node advertises:

```console
$ kubectl describe node <gpu-node> | grep nvidia.com/gpu
```

> [!NOTE]
> `Allocatable: 0` means the device plugin is missing or unhealthy, not that the GPU is busy. That points back to activation, not to your pod.

`Capacity` and `Allocatable` show the fixed slice count per node (normally 4, see [When a card is shared](#when-a-card-is-shared)). They do not shrink as pods claim slices, so a nonzero `Allocatable` does not mean a slice is free.

If the label, activation, and request all check out and the pod is still `Pending`, every slice on every GPU node is already claimed. Read the `Allocated resources` section further down in the same `describe` output, or list what already runs on the node:

```console
$ kubectl describe node <gpu-node>
$ kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
```

Add another GPU node, or free a slice by removing a pod.

## When a card is shared

More than one pod can land on the same card, and co-scheduled pods share its memory with no quota between them. So if a GPU job dies with a CUDA out-of-memory error, check what else is running on the card before you suspect the job. For how slicing and shared memory work, see [Share a GPU across pods](/docs/hetzner/apalla/workloads/specialized/share-a-gpu).

To keep a card to a single pod, give GPU servers their own pool and steer only that workload to it with a `nodeSelector` (Syself Autopilot does not support node taints). See [Add GPU nodes](/docs/hetzner/apalla/servers-and-nodes/provision/gpu-nodes) and [Node taints are not supported](/docs/hetzner/apalla/servers-and-nodes/scheduling/apply-taints).

## Collecting GPU logs

```console
$ kubectl get pods -A -o wide --field-selector spec.nodeName=<gpu-node>
$ journalctl -u gpu-activate.service -b
```

If a job runs slower than expected, or fails with a CUDA out-of-memory error, the `kubectl get pods` command above shows what else is scheduled on the card. A shared card is the usual cause.

From inside a GPU pod, `nvidia-smi` confirms the driver and card are visible.

## Related

- [Add GPU nodes](/docs/hetzner/apalla/servers-and-nodes/provision/gpu-nodes)
- [Run GPU workloads](/docs/hetzner/apalla/workloads/specialized/run-gpu-workloads)
- [Share a GPU across pods](/docs/hetzner/apalla/workloads/specialized/share-a-gpu)
- [Where to start troubleshooting](/docs/hetzner/apalla/support/where-to-start)