Skip to main content

Scale a node pool

Inspect 1.36

Scaling a pool is changing one number. Set the replica count and re-apply. Syself Autopilot adds or removes machines to match. If you want the number to follow demand instead, hand the pool to the . See Manual scaling vs the autoscaler below for how the two interact.

Change replicas and apply#

Edit the pool's entry under spec.topology.workers.machineDeployments:

yaml
		- class: workeramd64hcloud
  name: md-0
  replicas: 3
  replicas: 5
	

Apply the Cluster resource (or let GitOps sync it) and watch:

		$ kubectl get machines -w
	

What happens on scale-up#

Syself Autopilot creates new Machine objects (the Cluster API object that represents one server) up to the new count. On cloud, each provisions a fresh VM in a few minutes. On bare metal, each claims a free host from your inventory and installs onto it. A bare-metal worker node is ready in a few minutes.

Note

On bare metal, replicas set above the number of matching free hosts creates Machine objects that never reach their end state; they sit pending until a host is available. Set a bare-metal pool's replicas to the number of servers you have allocated for that cluster. Plan bare-metal servers per cluster, each server dedicated to one cluster, so a pool can always reach its target and the system does not sit in an unresolved state.

What happens on scale-down#

Syself Autopilot removes the surplus machines, draining each first:

Cordon #

The node is marked unschedulable.

Drain #

Pods are evicted with a grace period, respecting PodDisruptionBudgets (limits on how many pods of an app may be down at once). The drain waits for pods to leave within their budget, then proceeds anyway so a stuck PodDisruptionBudget cannot block the scale-down forever.

Remove #

On cloud, the VM is destroyed. On bare metal, the host is released back to the pool.

Syself Autopilot chooses which machine to remove. To remove a specific node instead, annotate its Machine object with cluster.x-k8s.io/delete-machine before you lower replicas. See for the full steps.

Tip

This cordon-drain-replace sequence is the same one Syself Autopilot uses for automatic self-healing. For how to protect a workload with PodDisruptionBudgets during any drain, see .

Bare-metal scale-down returns the host#

Scaling a bare-metal pool down does not destroy the server. The HetznerBareMetalHost returns to your inventory and can be claimed again later. The root disk is wiped on the next install, but data disks are left alone unless you .

Manual scaling vs the autoscaler#

For steady capacity, set replicas by hand. For workloads that rise and fall, hand the pool to the cluster autoscaler and do not set replicas in the Cluster object. The autoscaler is a controller that runs in your workload cluster: it watches for pending pods, and it has access to the management cluster, where it changes the MachineDeployment replica count directly. You give it a minimum and maximum as annotations, the guardrails it reads from the MachineDeployment. Annotations you put on the pool in the Cluster object are propagated to the real MachineDeployment.

yaml
		metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "10"
	
Warning

Do not set both. Cluster API rejects a pool that has replicas and the autoscaler min/max annotations at the same time.

See for the full setup.

Capacity planning#

Capacity planning is more than a node count. Think about CPU and memory, how many pods fit on one host (every node caps at maxPods: 220), the failure domain (how much you lose when one node or host goes down), and whether workloads should share nodes or stay separated.

A useful default: grow to about 10 worker nodes before you make the nodes bigger. At 10 nodes, losing one is 10% of the cluster, and you usually want 20 to 30% of capacity in reserve for bursts and for riding out a failure. Past 10 nodes, bigger machines often beat adding more small ones. The right shape depends on your workload, so treat this as a starting point.

A node's schedulable capacity (allocatable) is also less than its raw size, because the kubelet reserves CPU and memory for the system. Size pools against allocatable, and see the and for the numbers. To match spend to load, see .