Skip to main content

Run NUMA-aligned workloads

Inspect 1.36

On a multi-socket bare-metal server, memory has an address in more ways than one. Each socket owns a bank of RAM sitting right next to it, and reaching the other socket's RAM means crossing the interconnect between them. A pod whose threads run on socket 0 but whose memory landed on socket 1 pays that crossing on every access. The penalty is invisible until you profile it, and then it is everywhere.

NUMA alignment keeps a pod's CPUs and its memory on the same socket. Syself Autopilot does this automatically for pods that qualify, so a latency-sensitive database or real-time service does not need any special operator or annotation. It only needs to request its resources correctly.

flowchart LR
  subgraph S0["Socket 0 / NUMA node 0"]
    P["numa-app<br/>pinned to CPUs 8-11"]
    M0["local RAM"]
  end
  subgraph S1["Socket 1 / NUMA node 1"]
    M1["local RAM"]
  end
  S0 <-->|"interconnect (the tax)"| S1

Where it applies

Alignment is on by default for bare-metal workers and only matters on servers with two or more sockets. A single-socket machine has one NUMA node, so every pod on it is aligned by definition; Hetzner Cloud VMs are single-NUMA, and control planes never use it. Nothing to install, nothing to enable: a qualifying pod is aligned, and every other pod runs exactly as before.

Make a pod qualify

A pod is a candidate when both of these are true:

  1. Guaranteed QoS: every container sets requests equal to limits for both cpu and memory.
  2. Whole-integer CPU: the CPU value is a whole number (cpu: "4"), not a fraction (cpu: "3500m"). A fractional Guaranteed pod runs fine, but it draws from the shared CPU pool and gets no exclusive cores, so it cannot be pinned.
yaml
		apiVersion: v1
kind: Pod
metadata:
  name: numa-app
spec:
  # Force bare metal if the cluster also has cloud workers:
  # nodeSelector: { instance.hetzner.cloud/is-root-server: "true" }
  containers:
    - name: app
      image: your/image
      resources:
        requests: {cpu: "4", memory: 8Gi}
        limits: {cpu: "4", memory: 8Gi}
	

The Topology Manager aligns each container on its own, not the pod as a whole, so two containers in the same pod can land on different sockets. Size any single container to fit within one socket; a request larger than a socket cannot be aligned, though it still runs.

Note

Alignment is best-effort and never rejects a pod: it aligns when the pod fits inside a socket and admits it anyway when it does not. Exclusive whole cores leave the shared pool, so fewer pods fit onto the node; reserve this for workloads that measurably benefit. Once threads are pinned, the Linux kernel migrates the pod's memory pages to the socket they run on.

One pod, before and after

The difference shows up in one place: the CPUs the pod is allowed to run on. An unaligned pod is allowed the whole node and roams across both sockets. An aligned pod receives a small, fixed set of cores that all live on one NUMA node: the pod boxed inside socket 0 above. Same manifest, one value changed from 3500m to 4, and the kernel stops scattering it.

Verify pinning

Read the allowed CPU set from inside the pod:

		$ kubectl exec numa-app -- grep Cpus_allowed_list /proc/self/status
Cpus_allowed_list:  8-11
	

A short list such as 8-11 indicates a pinned pod; the full node range (0-63) indicates an unaligned one. To confirm those CPUs belong to a single NUMA node, SSH to the node and compare the list against /sys/devices/system/node/node*/cpulist and the kubelet's /var/lib/kubelet/cpu_manager_state. The pod's CPUs should fall entirely inside one node's cpulist.

The same alignment applies to latency-sensitive , which reserve their CPU and memory just like a Guaranteed pod.