Skip to main content

Spread nodes with placement groups

Inspect 1.36

Keep Hetzner Cloud VMs on separate physical hosts, so one hardware failure never takes down more than one node. A Hetzner Cloud spread placement group does this: Hetzner never puts two VMs from the same group on the same physical machine.

Without a placement group, Hetzner Cloud picks a physical host for each VM on its own. Two control-plane VMs could land on the same host. If that host fails, both VMs go down together, and the cluster can lose quorum (the majority of control-plane nodes that must stay up for the cluster to keep working). A worker pool without a group faces the same risk: one host failure can take out several of its nodes at once.

A placement group only decides which physical host a node runs on. It does not decide which node a pod runs on; for that, see .

Note

Placement groups apply to Hetzner Cloud VMs only. Bare-metal servers are already separate physical machines, so there is nothing to spread: skip placement groups for bare-metal pools.

Prerequisites#

  • Access to the management cluster (or the GitOps repository that syncs to it) and a Cluster object to edit.
  • A pool that runs on Hetzner Cloud VM nodes; see if you have not created one.

Configure the groups#

Declare the groups #

Add hcloudPlacementGroups to spec.topology.variables. Hetzner Cloud supports only the spread type, and a spread group holds at most 10 servers (see the Hetzner documentation⁠).

yaml
		- name: hcloudPlacementGroups
  value:
    - name: myPlacementGroup1
      type: spread
    - name: myPlacementGroup2
      type: spread
Note

The name you set here is not the name Hetzner shows. In the Hetzner Cloud console the group appears prefixed with the cluster name, as <cluster>-myPlacementGroup1. Names are scoped per cluster, so two clusters in the same Hetzner project can safely reuse the same name: each cluster gets its own separate group, and the 10-server cap applies per cluster, not across the project.

Assign one to the control plane #

Set controlPlanePlacementGroupNameHcloud to one of the names from step 1. This protects quorum by keeping control-plane nodes off the same host; see for control-plane sizing.

yaml
		- name: controlPlanePlacementGroupNameHcloud
  value: myPlacementGroup2

Assign per-pool groups for workers #

Set workerMachinePlacementGroupNameHcloud in a worker pool's variables.overrides. Give each pool its own group so a host failure costs at most one node per pool:

yaml
		machineDeployments:
  - class: workeramd64hcloud
    name: md-0
    replicas: 5
    failureDomain: nbg1
    variables:
      overrides:
        - name: workerMachineTypeHcloud
          value: cpx42
        - name: workerMachinePlacementGroupNameHcloud
          value: myPlacementGroup1
Important

A spread group holds at most 10 servers, so a worker pool that uses one is capped at 10 nodes. That fits a control plane (three or five nodes) and small worker pools well. A pool that needs to grow past 10 either runs without a group, or splits into several pools of up to 10, each with its own group.

Tip

Run several worker pools and give each its own group. Each group spreads independently, so a pool of compute-dedicated (ccx) nodes can have its own group, separate from your general-purpose worker pool.

Variable reference#

Variable Type Required Default
hcloudPlacementGroups array of objects (name, type) no none
controlPlanePlacementGroupNameHcloud string no none
workerMachinePlacementGroupNameHcloud string no none

See the for every variable a cluster supports.

Complete example#

Placement groups lower the odds of losing more than one node at once. Self-healing handles whatever a host failure still takes down: it detects the unhealthy node and replaces it, see .

Add GPU nodes

Run a GPU worker pool on Hetzner bare metal; the driver is already on the node, so GPU pods schedule with no setup.

Node pools and machine deployments

A pool is one entry in your Cluster resource, a class, a machine type, a count, and optional labels.