Node and hardware metrics

node-exporter gives you the per-node hardware and OS metrics: CPU, memory, filesystem, disk, and network. You install it yourself, as a DaemonSet on the host network, and on a Syself Autopilot cluster its metrics stay on the node, so a ServiceMonitor does not reach it. Scrape it over loopback instead.

Install it#

The chart's defaults are already what this platform needs: hostNetwork: true, a NoSchedule toleration so it lands on control-plane nodes too, and no ServiceMonitor. There is no values file:

		$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install node-exporter prometheus-community/prometheus-node-exporter \
  --namespace monitoring \
  --version <chart-version>
	

Pin --version to a known chart release so an update does not change behavior under you. Confirm you get one pod per node, control planes included:

		$ kubectl -n monitoring get ds node-exporter-prometheus-node-exporter
NAME                                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
node-exporter-prometheus-node-exporter   4         4         4       4            4           kubernetes.io/os=linux   1m
	

A node missing from that count is a node whose hardware you are not watching.

Why the loopback scraper#

node-exporter's DaemonSet listens on the node, and Syself Autopilot cluster nodes carry public IPs, so the host firewall does not let an arbitrary Prometheus pod reach it. That is why its ServiceMonitor stays off, which is the chart's default, and the System Alloy scrapes 127.0.0.1:9100 instead. Because node-exporter is your install rather than something the platform ships, that job is the one block in the agent's config to delete if you skip this page. Confirm it is listening on a node before chasing an error:

		$ ss -tlnp | grep 9100
	

What you get#

The usual node-exporter series, per node, for both cloud and bare-metal pools:

• CPU by mode, memory used and available, load average.
• Filesystem free space and inodes per mount, which is how you catch a /var filling with logs before it becomes a problem.
• Disk I/O per device, including NVMe, useful under a TopoLVM local-storage workload where disk pressure on one node hurts the pods pinned to it.
• Network bytes, errors, and drops per interface.
• On bare metal, hardware sensor readings where the board exposes them, such as component temperatures (node-exporter's hwmon collector).

Label the series by pool (for example a pool relabel from the node's pool label) so you can compare a bare-metal database pool against a cloud burst pool in one view.

What the health daemon already covers#

node-exporter is for graphs and trends. The node health daemon does something different: it detects the disk and hardware problems that matter. It sets conditions such as DisksFailure, DiskWearHigh, DiskTemperatureHigh, and DiskUsageHigh on the Node, and it records a disk inventory, including wear level, in the autopilot.syself.com/disks node annotation, so you get that hardware picture without setting anything up.

A disk condition is for you to alert on, not something the platform fixes on its own. The daemon does not reboot or replace a node over a failing disk, on cloud or bare metal, because neither would fix the hardware. Use node-exporter to watch trends and capacity over time, and alert on the daemon's disk conditions (see Platform alert rules ) to catch a failing disk on any pool.

GPU metrics with the NVIDIA DCGM exporter#

On GPU worker nodes, NVIDIA's Data Center GPU Manager (DCGM) exporter reports GPU utilization, VRAM use, power, and temperature. Install its Helm chart pinned to the GPU pool, and give the pod one GPU so the exporter can read the hardware:

		$ helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
$ helm upgrade -i nvidia-dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace monitoring \
  --set serviceMonitor.enabled=false \
  --set-string nodeSelector."autopilot\.syself\.com/gpu"="true" \
  --set "resources.limits.nvidia\.com/gpu=1"
	

The exporter runs as a DaemonSet on the GPU nodes and serves metrics on port 9400 (service port name metrics) on the pod network, so it is scraped like an application rather than like a platform component: declare a ServiceMonitor for it and the Application Alloy picks it up (Custom application metrics ). Requesting nvidia.com/gpu=1 triggers containerd's Container Device Interface hook to mount the /dev/nvidia* devices and driver libraries into the pod. Because Syself Autopilot enables GPU time-slicing by default, the exporter takes one virtual time-sliced slot and your workloads keep running on the same physical GPU alongside it.

The core series to graph are DCGM_FI_DEV_GPU_UTIL (compute utilization), DCGM_FI_DEV_FB_USED (VRAM used, in MB), and DCGM_FI_DEV_GPU_TEMP (temperature).