Every Syself Autopilot cluster runs live monitoring but keeps no history. Out of the box it runs metrics-server for CPU and memory, Hubble for network flows, a node health daemon that reports node problems as Kubernetes conditions, and audit logs on every node. Long-term metrics, dashboards, log aggregation, and alerting are yours to run, on storage you own.

Most managed platforms store your metrics and logs in their own backend. They control retention and access, and if you leave, the history stays in their account.

## What Syself ships and what you bring

Syself builds in the live monitoring. You run retention, dashboards, and alerting.

> [!NOTE]
> For step-by-step setup guides for Prometheus, Grafana, Loki, Tempo, and Alloy, see the dedicated [Observability documentation](/docs/hetzner/apalla/observability/overview).

| Syself ships in every cluster                                                                                                                                                          | You bring (not installed by default)                                  |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| metrics-server, reading CPU and memory from every node and serving it through the Kubernetes metrics API                                                                               | Collection: an agent on each node, plus one for your applications     |
| Hubble, recording every connection and every dropped packet in Cilium                                                                                                                  | Long-term metrics storage: Prometheus, Grafana Mimir, VictoriaMetrics |
| `syself-agent`, writing NodeConditions and JSON health annotations                                                                                                                     | Dashboards: Grafana. Log aggregation: Loki. Tracing: Tempo or Jaeger  |
| Audit logs on each node: host `auditd`, the Kubernetes API audit log, and the audit log of KubeGate (Syself's gate in front of the API server, admitting or rejecting each connection) | Alerting: Prometheus Alertmanager                                     |

No Prometheus-Operator CRDs ship either. Install kube-prometheus-stack and the `ServiceMonitor` objects are yours to write.

Monitoring is built in because it must run live inside the cluster, connect to remediation, and keep working across upgrades. You run retention because you are the one who reads the history later, after a problem, including after a failed node has been replaced.

## What the built-in monitoring does

metrics-server reads CPU and memory from every node on a short cycle. It serves the result through the `metrics.k8s.io` API. `kubectl top` reads from it, and so does any CPU- or memory-based HorizontalPodAutoscaler. Without it an HPA cannot scale, and it logs `unable to fetch metrics from resource metrics API`.

metrics-server holds only the latest sample per pod and node. For history you run Prometheus, a common tool that collects and stores metrics over time.

`syself-agent` runs on every node. It writes NodeConditions onto the Node object, and those conditions drive remediation: on cloud pools a failed node is replaced, on bare-metal pools most conditions alert a human instead. [Self-healing and node replacement](/docs/hetzner/apalla/concepts/operations/self-healing-and-node-replacement) covers which conditions do what and why the two pools differ; [node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) lists them all.

Alongside the conditions, the daemon writes one JSON health report per domain as annotations you can read straight off the node: `autopilot.syself.com/storage`, `/disks`, `/certs`, `/tamper`, and `/services`.

```console
$ kubectl get node <name> \
  -o jsonpath='{.metadata.annotations.autopilot\.syself\.com/disks}' | jq .
```

Hubble records every pod-to-pod and pod-to-external connection and every network-policy verdict in Cilium. It is on by default, and the history is kept only in memory.

## Why some metrics need a host-network DaemonSet

Metrics are the numbers a component reports about itself, like CPU use or request counts. Monitoring tools scrape them: on a schedule they read those numbers from each component. Where a component can be scraped from depends on how the cluster is wired.

Many of the metrics endpoints on a node are plain HTTP with no login: etcd, KubeGate, the health daemon, and all of Cilium's. An unauthenticated metrics page leaks hostnames, environment variables, and credentials to anyone who can open it. So every metrics listener on the node's network binds to the loopback address, the local-only address that only the machine itself can reach. There is no network route to them to secure, on any topology.

The host firewall already drops unsolicited inbound traffic, and Cilium's clusterwide policy denies traffic by default in each direction (see [Networking](/docs/hetzner/apalla/concepts/internals/networking) for why public-IP nodes are safe). Loopback is a second layer of protection, independent of the firewall. If a firewall rule is ever widened or fails to apply, a listener on the loopback address is still unreachable from off the node.

Which machine the loopback address refers to depends on the network space of the process reading it. A monitoring pod like Prometheus has its own network space, so its loopback is the pod, not the node. A `ServiceMonitor` scrapes from that pod, so it can never reach a loopback endpoint on the host. A DaemonSet works instead: it runs on each node's own network. Its loopback is the node's. It reads the local endpoints on the node and forwards the data to your Prometheus.

```mermaid
flowchart LR
  subgraph node["One node"]
    ep["Host-network metrics endpoints<br>etcd, KubeGate, Cilium, syself-agent"] -->|"bind to loopback only"| lo["Loopback<br>node-local address"]
    ds["Scrape agent pod<br>runs on the node's network"] -->|"reads local endpoints"| lo
  end
  prompod["Prometheus pod<br>own network space"] -.->|"cannot reach the node's loopback"| lo
  inet["Internet"] -.->|"cannot reach loopback"| lo
  ds -->|"forwards metrics"| prom["Your Prometheus / Mimir"]
  prom --> g["Grafana, Alertmanager"]
```

Which components sit on the pod network (scrapable with an ordinary `ServiceMonitor`) and which sit on the host network (reachable only by the DaemonSet) is listed in [metrics per component](/docs/hetzner/apalla/observability/reference/metrics-per-component); the full inventory of what binds where is in [Ports and listeners](/docs/hetzner/apalla/reference/ports-and-listeners). Opening a metrics port on the host firewall is never the answer: a node-local DaemonSet never crosses the firewall, so there is no rule to add.

That DaemonSet collects infrastructure telemetry only. Your applications' metrics go to a separate, horizontally scaled collector, because one careless high-cardinality label, a request ID for example, could turn a single series into hundreds of thousands and take the node's kubelet, etcd, and log shipping down with it. [Using Alloy for observability](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) covers how the two collectors divide the work.

## Audit logs on each node

Every cluster writes security-audit logs with no logging stack installed. Three signals make up the trail: the host `auditd` log on every node, the Kubernetes API audit log on each control plane, and KubeGate's allow/deny log at the API server's front gate. They land on the writable `/var` partition and rotate locally, with the `auditd` ruleset scoped for the logging and monitoring controls of BSI C5:2020 and locked against tampering by a root user. That helps your audit, but it is not a certification Syself holds. [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) covers the signals, their local caps, and the real-client-IP attribution in full.

> [!WARNING]
> Everything under `/var` is lost when a node is reprovisioned. Syself Autopilot replaces nodes rather than repairing them, so a node that fails takes its local logs and audit trail with it. Local retention is a short-term buffer of days to weeks, not the system of record. Ship logs off-node from day one, into your own log store or a SIEM (the central system where a security team keeps and searches security events); the pipeline is yours to run. See [Collect logs](/docs/hetzner/apalla/observability/logs/collect-container-and-journald-logs).

The OS is immutable (fixed at build time, so nothing can be added to a running node) and has no package manager, so a log agent cannot be installed onto the node. It runs as a DaemonSet with host-path mounts instead: the container logs, the systemd journal, and the three audit logs above. Reading the `auditd` log needs a privileged pod, because the file is root-owned and readable only by root.

## What Syself does not monitor

**GPU metrics.** GPU nodes export no GPU metrics as shipped. The NVIDIA device plugin publishes none of its own, node-feature-discovery is disabled, and the DCGM exporter is not bundled. Beyond the usual kubelet and node-health series a GPU node shows nothing: no utilization, no GPU memory, no temperature. Deploy the DCGM exporter yourself.

**Cilium Envoy metrics.** These are disabled. The upstream chart binds that listener to a public, internet-facing address, which would publish it on the node's public IP, so the platform turns it off rather than break the loopback rule. Envoy does no L7 work here anyway, so the lost series are near-empty.

**Hubble history.** Flow history lives in memory, so it is gone when the agent restarts or the node is replaced. For a record you can query later, turn on Hubble flow export with a `CiliumNodeConfig`. That object overrides the agent's config without touching the platform-managed ConfigMap, so an upgrade does not undo it. See [Alert on dropped packets](/docs/hetzner/apalla/observability/network-flows/alert-on-drops).

**Off-node shipping.** Nothing leaves the cluster by default, and nothing sends telemetry to Syself. The retention pipeline, its storage, and its access control are yours to build.

## Build your pipeline in this order

Send off-node first the data that exists only on the node: the audit trail and container logs. A node that fails takes them with it, so ship them first. The same node-level agent then carries the metrics, so one deployment covers both. Add the application collector when you have workloads to watch, and dashboards last. `kubectl top` and the health annotations give you live triage, but you only have history if you started storing it earlier.

## Related pages

- [Networking](/docs/hetzner/apalla/concepts/internals/networking) covers how a node is addressed and why each one runs its own firewall.
- [Backup and disaster recovery](/docs/hetzner/apalla/concepts/operations/backup-and-disaster-recovery) covers the retention and storage you own.
- [Ownership and shared responsibility](/docs/hetzner/apalla/concepts/ownership/ownership-and-shared-responsibility) draws the line between what Syself runs and what you run.
- [Using Alloy for observability](/docs/hetzner/apalla/observability/collection/using-alloy-for-observability) covers the collectors that read all of this, and why there are two of them.