Deploy the System Alloy

The System Alloy is the one agent that runs on every node, and it collects everything the node itself knows: the control-plane metrics on loopback, the kubelet and cadvisor series, the Cilium and cluster component endpoints, and the container, systemd, and audit logs. One Helm release covers all of it. Application metrics are deliberately absent, for the reason in Using Alloy for observability .

Prerequisites: a metrics backend that accepts remote-write and a Loki that accepts pushes, both reachable from inside the cluster (Set up Prometheus and Set up Loki ), plus Helm and kubectl access.

What the config has to solve

Four things make this config specific to Syself Autopilot rather than a stock Kubernetes install.

1. Loopback binding. Most platform metrics endpoints bind to 127.0.0.1, so the agent shares the node's network namespace with hostNetwork: true and dials loopback directly. dnsPolicy: ClusterFirstWithHostNet keeps cluster DNS resolvable from there, which the backend Service names need.

2. A per-node identity. Every agent scrapes the same local address, 127.0.0.1:<port>, so by default every node's metrics carry the same instance label. The backend then cannot tell the nodes apart, treats them as one series, and rejects the mixed samples as out of order (err-mimir-sample-out-of-order on Grafana Mimir). The config writes the node name into the instance and node labels instead, using the Downward API.

3. Node-role gating. etcd, kube-controller-manager, kube-scheduler, KubeGate, and syself-tunnel-server run only on control planes. syself-proxy runs only on workers, and the CSI node plugin only on cloud workers. If every node gets the same target list, each node also tries to scrape components that do not run on it. Those scrapes fail with connection refused forever, and a TargetDown alert fires that never clears. The config reads the local node's labels and keeps each job only where that component runs. The labels the gates read: control-plane nodes carry node-role.kubernetes.io/control-plane with an empty value, so the gate checks that the label exists rather than what it says. Workers carry node.kubernetes.io/worker=true, and the machine type is in autopilot.syself.com/machine-type, hcloud or baremetal. The CSI node plugin is the one job that needs two labels: it runs only on cloud workers, and gating it on the worker label alone would include bare-metal workers, where that port never listens.

4. Self-signed certificates on loopback. A TLS client checks the address it dialled against the names in the server's certificate. kube-controller-manager and kube-scheduler use self-signed certificates, and the kubelet's certificate names the node's IP, not 127.0.0.1, so scraping them over loopback fails verification either way. Those jobs set insecure_skip_verify, which is safe here because the connection never leaves the node. The API server is the exception: its certificate does include 127.0.0.1, so that job verifies against the ServiceAccount CA bundle instead of skipping.

Deploy it

		$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm upgrade -i system-alloy grafana/alloy \
  --namespace monitoring --create-namespace \
  --version <chart-version> \
  --values system-alloy-values.yaml
	

		$ kubectl -n monitoring get pods -l app.kubernetes.io/instance=system-alloy -o wide
	

		$ kubectl -n monitoring port-forward daemonset/system-alloy 12345:12345
	

		count(up{job="etcd"}) by (instance)
count(up{job="cilium-agent"}) by (instance)
	

Add the Hubble flow log

If you turned on Hubble flow export (Alert on dropped packets ), the verdict log is a file on the node like any other, and the values file above already mounts its directory. Add a source for it:

		local.file_match "hubble_flows" {
  path_targets = [{
    __path__ = "/var/run/cilium/hubble/events.log",
    job      = "hubble-flows",
    node     = sys.env("NODE_NAME"),
  }]
}
 
loki.source.file "hubble_flows" {
  targets    = local.file_match.hubble_flows.targets
  forward_to = [loki.write.central.receiver]
}
	

Keep the log volume down

The values file already filters the systemd journal: its discovery.relabel "journal" block keeps log lines from kubelet, containerd, and the syself-* services and drops every other unit, which is most of the volume for none of the value. Container logs are the other large contributor and vary entirely with your workloads, so drop or sample the high-volume namespaces rather than paying to store what you will not query.

When Loki is slow or an incident floods logs, Alloy's disk buffer at storagePath absorbs the burst. Size that volume so a spike degrades into delay instead of taking the node's memory with it. Log retention and sizing has the per-node estimates to size against.

With the node signals flowing, add the Application Alloy for your workloads' metrics and traces. For what the series you just started collecting actually tell you, see Control-plane metrics and Data-plane and cluster component metrics . If a target will not come up, Troubleshoot the collectors covers the failures worth knowing by name.