Node resources and limits

Every node has a hard limit on how many pods it runs. Below that limit, a slice of CPU and memory is reserved for the OS and system processes, so pods never starve the system. Before the node runs out of memory or disk, the kubelet (the Kubernetes agent on each node) removes pods to protect node stability.

Pod limit#

Syself sets maxPods: 220 in the kubelet configuration. The kubelet refuses the 221st pod.

The effective workload capacity is a bit lower than 220. A few of the slots go to per-node DaemonSet pods that Syself runs on every node. The Syself node daemons run as systemd services, not pods, so they cost nothing from the pod limit.

The pod limit must not exceed the number of usable IP addresses on the node's pod subnet. Each pod gets one IP from the node's slice of the cluster pod CIDR (100.64.0.0/11). The controller manager carves that CIDR into per-node blocks. Syself sets the per-node mask to /24 (256 addresses, about 254 usable) with the controller manager's --node-cidr-mask-size, so maxPods: 220 sits just under the per-node IP budget. Against the /11 pod CIDR that allows 8192 nodes. It is not a cluster variable and cannot be changed: a node's pod subnet is fixed once assigned, so the per-node mask cannot change on an existing cluster.

Why a fresh node shows less than its hardware#

Run kubectl describe node on an idle node and you see two numbers for CPU and memory:

• Capacity: the raw hardware: all the CPU cores and all the RAM.
• Allocatable: what the scheduler may hand to pods. This is always less than Capacity.

The gap exists because a node does not run only pods. A node also runs the OS and system daemons, and none of them are pods, so the scheduler cannot see or account for them. These daemons include systemd, sshd, ntpd-rs (the time sync daemon), auditd, containerd (the container runtime), one shim process per running container, the kubelet, and the Syself node daemons. If pods could claim the whole machine, a burst of load would starve the system daemons. The kubelet would miss its heartbeat, the node would flip to NotReady, and containerd would stop starting or stopping containers.

The kubelet prevents this with reserved capacity that the scheduler never allocates:

		Allocatable = Capacity - systemReserved - evictionHard
	

The systemReserved slice is described below. The evictionHard thresholds add another buffer on top, described in the next section.

CPU and memory reservation#

Syself's systemReserved baseline is:

		systemReserved:
  cpu: 100m
  memory: 300Mi
  ephemeral-storage: 1Gi
  pid: "8192"
	

These are not what runs. When each node joins the cluster, Syself reads the node's actual CPU count and total RAM, then rewrites the cpu and memory fields with values sized for that node. The ephemeral-storage and pid fields are preserved from the baseline.

Syself uses a tiered memory curve and adds an allowance for the Syself node daemons that run in system.slice. The resulting reserved memory is roughly:

The larger the node, the more memory is reserved, but the reserved fraction shrinks as a percentage of total RAM. This tiered approach matters because the fleet spans small cloud VMs and large bare-metal servers: a single fixed value would either over-reserve on small nodes or under-protect on large ones.

This is the same shape managed Kubernetes uses across the industry: a reservation that scales with node size, often tied to the number of pods a node can run, with a small hard-eviction threshold held back on top. The reason is the same everywhere: the kubelet, the container runtime, and the OS daemons need guaranteed headroom, or a burst of workload memory takes the node down instead of a single pod.

Eviction thresholds#

evictionHard thresholds trigger immediate pod eviction (no grace period) when a node is under pressure. They protect node stability by shedding workload before the node reaches a critical state.

The pid.available threshold evicts pods before the node runs out of process IDs (PIDs), the identifiers Linux assigns to running processes.

When any threshold is crossed, the kubelet evicts pods in priority order, lowest-priority first. System-critical pods (those with priorityClassName: system-node-critical or system-cluster-critical) are the last to be removed. After eviction, the kubelet does not reschedule the removed pods locally; the scheduler places them elsewhere.

The nodefs thresholds protect the writable DATA partition mounted at /var, where kubelet and containerd keep their state. The OS root itself is a sealed read-only dm-verity image and is never written. There is no separate image disk, so imagefs resolves to the same partition. Running close to either threshold causes eviction pressure, which shows in kubectl describe node under Conditions.

Where each setting lives#

The cluster pod and service CIDRs are fixed at cluster creation. You cannot change them on a running cluster. If you need to run more nodes or more pods per node than the defaults allow, set wider CIDRs on the Cluster object before you create it.

Related#

• Cluster variables reference
• Node health conditions
• Server types and sizing
• Create a cluster : where the pod and service CIDRs are set, once, at creation.
• Steer workload placement