Set up Grafana

Inspect 1.36

Grafana is where you view your metrics and logs as graphs and dashboards. It is its own chart, and it reads the store you set up in Set up Prometheus over the cluster network. Declare that connection in the values rather than clicking it in later, so a reinstalled Grafana comes back knowing where its data is.

Install it #

grafana-values.yamlyaml

		# A data source is a backend Grafana reads from to draw its graphs. Declared
# here, they survive a reinstall; added in the UI, they do not.
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        access: proxy
        url: http://prometheus-operated.monitoring.svc.cluster.local:9090
        isDefault: true
# Load dashboards from labelled ConfigMaps, and let each one pick its folder.
sidecar:
  dashboards:
    enabled: true
    label: grafana_dashboard
    folderAnnotation: grafana_folder
    provider:
      foldersFromFilesStructure: true
# standard volumes provision only on cloud nodes, so keep Grafana there.
# Change this if you point storageClassName at a local class instead.
nodeSelector:
  autopilot.syself.com/machine-type: hcloud
persistence:
  enabled: true
  # standard = Hetzner Cloud volumes, cloud nodes only. The nodeSelector above
  # keeps this on a cloud node. To run on a bare-metal node instead, drop it and
  # name the local class matching that node's disks (local-nvme, local-ssd, or
  # local-hdd through TopoLVM).
  storageClassName: standard
  size: 10Gi

		$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm install grafana grafana/grafana \
  --namespace monitoring \
  --version <chart-version> \
  --values grafana-values.yaml

Pin --version to a known chart release so an update does not change behavior under you.

Note

The nodeSelector is what keeps the standard claim provisionable. On a cluster with bare-metal nodes, a Grafana scheduled onto one leaves its PVC Pending with no topology key found and the pod never starts. If that has already happened, adding the selector is not enough on its own: the claim keeps a volume.kubernetes.io/selected-node annotation pointing at that node, so delete the PVC and helm upgrade to have the chart recreate it. To run Grafana on a bare-metal node instead, name the local class matching that node's disks.

For a single operator, use a port-forward and read the admin password from the secret the chart created. A port-forward opens a private connection from your machine to the service, so Grafana stays off the network:

		$ kubectl -n monitoring port-forward svc/grafana 3000:80
$ kubectl -n monitoring get secret grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d && echo

Then open http://localhost:3000. Under Connections, the Prometheus data source should already be there; Save & test on it reports Successfully queried the Prometheus API.

Add the other data sources as you install them #

Add each backend to the same datasources list so metrics, logs, and traces share one query surface:

Loki (http://loki-gateway.monitoring.svc.cluster.local) for logs, once you set up Loki .
Tempo for traces, once you install it .

grafana-values.yamlyaml

		datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Loki
        type: loki
        access: proxy
        url: http://loki-gateway.monitoring.svc.cluster.local

Log a team in with OIDC #

For anything beyond one operator, connect Grafana to your identity provider so access follows your existing accounts and groups. OIDC is a standard login method that lets people sign in with those accounts instead of a shared password. Configure your IdP settings under the grafana.ini.auth.generic_oauth key in your Helm values. To avoid handing out the built-in admin password, use the role_attribute_path setting to automatically map an IdP group (like grafana-admins) to the Grafana Admin role. For the full list of configuration options, see the Grafana Generic OAuth documentation⁠.

Warning

Do not expose Grafana on a type: LoadBalancer Service with only the default admin login. It shows your whole cluster's telemetry. For a team, put it behind an ingress with the OIDC login above.

The persistence block above keeps the dashboards you build by hand, its users, and its settings across a pod move. It is still better to keep dashboards in Git, so the pod's state does not matter: Dashboards as code .

Next, load dashboards that match a Syself Autopilot cluster's real components: Dashboards for a Syself Autopilot cluster .

Correlate traces, metrics, and logs

Link a slow trace to the exact log lines and metric spikes behind it, so one Grafana click moves from symptom to cause.

Dashboards for a Syself Autopilot cluster

The generic Kubernetes dashboards miss the loopback and Cilium metrics, so map the panels that actually reflect a Syself Autopilot cluster.