- Docs
- Observability
Observability
What ships in every cluster, and how to collect metrics, logs, traces, dashboards, and alerts on top
- Overview What every Syself Autopilot cluster ships for observability, and the metrics, dashboards, logs, traces, and alerts you run yourself on top.
- What is shipped by default and what you bring metrics-server, Hubble, and the node health daemon are built in; Prometheus, Grafana, Loki, tracing, and alerting are not.
- Detection vs retention The health daemon detects node problems in real time and drives self-healing; your observability stack retains the evidence so you can investigate after a node is replaced.
- Plan your observability stack Decide in-cluster versus central, how long to keep data, and how big it will get, before you install anything.
Collection
- Using Alloy for observability Run one Grafana Alloy DaemonSet for infrastructure telemetry and one clustered Alloy StatefulSet for application telemetry, so a runaway application cannot take your node telemetry down with it.
- Deploy the System Alloy Install one Grafana Alloy DaemonSet on the host network that scrapes every loopback and cluster component metrics endpoint and ships container, journald, and audit logs off the node.
- Deploy the Application Alloy Run a clustered Grafana Alloy StatefulSet that scrapes your workloads through their existing ServiceMonitors and receives OTLP traces, sharded across replicas.
- Troubleshoot the collectors The failure modes worth knowing by name, from a target that is down and a series scraped twice to out-of-order samples and a cardinality spike.
Metrics
- metrics-server and live usage metrics-server ships in every cluster and feeds kubectl top and the HorizontalPodAutoscaler, but it keeps only the latest sample, so it is not history.
- Set up Prometheus Install the Prometheus Operator and one Prometheus that only receives, because the Alloy collectors do all the scraping.
- Control-plane metrics etcd, kube-controller-manager, kube-scheduler, the kubelet, and KubeGate keep their metrics on loopback, and this page covers what each one tells you and how it is reached.
- Cluster object metrics with kube-state-metrics kube-state-metrics turns every Kubernetes object's state into a metric, including the custom NodeConditions the health daemon latches, so you can alert on them.
- Node and hardware metrics node-exporter binds to the host, so scrape it over loopback to get CPU, memory, disk, NVMe, and network metrics for cloud and bare-metal pools.
- Custom application metrics Expose your app's metrics and declare a ServiceMonitor or PodMonitor; the Application Alloy reads those objects, shards the targets across replicas, and keeps app cardinality away from node telemetry.
- Long-term storage and remote-write Prometheus local storage is short-lived, so remote-write to Thanos, Mimir, or Victoria Metrics for durable, queryable, multi-cluster history.
- Data-plane and cluster component metrics The platform's networking, DNS, storage, and cloud-integration cluster components each report their own metrics, some on loopback and some on the pod network, and this page covers what each one tells you.
Reference
- Metrics per component Every Syself Autopilot component's metrics endpoint, its port and scheme, which nodes it exists on, and whether it sits on the pod network or the node's loopback. Health-daemon metrics and how to scrape a host-network node.
- Node health conditions Every NodeCondition the health daemon can set, what triggers each one, how to read them and the health report on a node, and how to alert on them with kube-state-metrics.
Logs
- Log sources on a sealed node A sealed Syself node has a read-only root and a writable /var, and this page lists every log stream that lives there and disappears on reprovision.
- Set up Loki Install Loki as the log store, back it with object storage, and add it to Grafana so logs and metrics share one query surface.
- Collect container and journald logs Container stdout and the systemd journal are the two operational log streams on every node, and this page covers what each holds, how they are labelled, and how to cut the volume.
- Ship audit logs off-node The auditd, Kubernetes API audit, and KubeGate audit streams are your evidence trail, so ship all three off-node before a replaced node takes them.
- Log retention and sizing On-node logs are bounded and rotate, so this page gives the caps you start from and how to size retention in your own store.
Traces
- Ingest OTLP traces Your services export spans in OTLP to the Application Alloy, which batches them, derives span metrics, samples what you keep, and forwards to Tempo or Jaeger.
- Install Tempo or Jaeger Pick a trace backend, Tempo for object-storage-cheap traces beside Loki, or Jaeger for a standalone UI, and wire it to the collector.
- Instrument a workload Add OpenTelemetry to your service, with SDK code or zero-code auto-instrumentation, and point it at the collector so traces start flowing.
- Correlate traces, metrics, and logs Link a slow trace to the exact log lines and metric spikes behind it, so one Grafana click moves from symptom to cause.
Dashboards
- Set up Grafana Install Grafana, declare Prometheus and Loki as data sources in its values, reach it safely, and log a team in with OIDC.
- Dashboards for a Syself Autopilot cluster The generic Kubernetes dashboards miss the loopback and Cilium metrics, so map the panels that actually reflect a Syself Autopilot cluster.
- Dashboards as code Keep dashboards in Git and let the Grafana sidecar or Argo CD provision them, so every client cluster gets the same views without click-ops.
Alerting
- Set up Alertmanager Deploy Alertmanager as its own object, hold its config in a Secret, load rules with PrometheusRule objects, and confirm an alert flows end to end.
- Alert routing and receivers Route alerts by severity, team, and cluster to Slack, PagerDuty, email, or a webhook, so the right person is paged and clients are separated.
- Platform alert rules Some conditions never trigger self-healing on bare-metal pools, so alerting is the only way to act on node integrity and capacity, and this page ships those rules.
- SLOs and error budgets Define service-level objectives on your own metrics and alert on burn rate instead of raw thresholds, so pages mean a real customer problem.
- Silences and grouping Group related alerts into one notification and silence known noise during maintenance, so on-call is not buried when a node pool rolls.
Network flows
- See flows with Hubble Hubble is on by default and records every connection and drop, so scrape its metrics and use the CLI to watch flows live.
- Alert on dropped packets A rise in dropped packets means a workload is being blocked or probed, so alert on hubble_drop_total and keep a queryable flow log.
- The Hubble UI The Hubble UI draws the live service map but has no login, so reach it by port-forward or put it behind authenticated ingress before anyone shares it.
Fleet and management cluster
- Monitor cluster and machine health Syself runs the management cluster and heals your clusters, but the Cluster API objects it reconciles are yours to watch, so alert on their status.
- Multi-cluster observability Run one Grafana, one long-term store, and one alerting plane over many clusters so an agency operates a whole fleet from a single view.