How to Run GPU workloads

Introduction#

Syself Autopilot supports Nvidia GPUs in Kubernetes clusters running on bare metal servers from Hetzner. You can run AI, machine learning, inference, and accelerated compute workloads directly inside Kubernetes. Typical workloads that benefit from this feature include:

• LLM inference
• AI agents and automation
• Machine learning training
• Vector databases
• GPU-accelerated data processing
• Computer vision workloads
• Media transcoding and rendering

Provisioning GPU servers#

To use GPUs with Syself Autopilot, first order bare metal servers with Nvidia GPUs from Hetzner.

Register them into your cluster following the bare metal onboarding guide .

After the servers join the cluster, Syself Autopilot automatically detects supported Nvidia GPUs and configures the required drivers and Kubernetes components.

Automatic Nvidia GPU driver installation#

Autopilot automatically installs and manages the required Nvidia components for GPU workloads, including:

• Nvidia GPU drivers
• Kubernetes device plugins
• GPU runtime configuration
• Node-level GPU discovery

This removes the need to manually SSH into nodes and do the configuration by hand.

Running GPU workloads in Kubernetes#

After GPU nodes are available, applications can request GPU resources directly through standard Kubernetes manifests.

GPU resources are exposed through the nvidia.com/gpu resource type. Kubernetes schedules workloads only onto nodes with available GPU capacity.

Example:

		apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1
	

This example requests one Nvidia GPU for the container.

Once the pod is running, Kubernetes automatically schedules it onto a GPU-enabled node.

Verifying GPU availability#

To verify that GPUs are available in your cluster, inspect the node capacity:

		$ kubectl describe node <node-name>
	

You should see GPU resources exposed under the node capacity and allocatable sections:

		Capacity:
  nvidia.com/gpu: 1
	

You can also verify GPU access from inside a container using:

		$ nvidia-smi
	

Example AI and machine learning workloads#

GPU-enabled Kubernetes clusters can run a wide range of AI and accelerated compute applications, including:

• LLM inference with vLLM or Ollama
• PyTorch and TensorFlow training workloads
• Ray distributed compute workloads
• Jupyter notebooks
• Vector databases
• Computer vision pipelines
• Video encoding and transcoding

Best practices#

For production GPU workloads, we recommend:

• Using dedicated GPU machine deployments
• Separating inference and training workloads, if possible even in dedicated clusters
• Monitoring GPU utilization
• Using node labels for workload isolation
• Defining GPU resource limits explicitly in workloads

If you need assistance in managing GPU-enabled clusters, we are here to help!