Deploy the System Alloy

Inspect 1.36

The System Alloy is the one agent that runs on every node, and it collects everything the node itself knows: the control-plane metrics on loopback, the kubelet and cadvisor series, the Cilium and cluster component endpoints, and the container, systemd, and audit logs. One Helm release covers all of it. Application metrics are deliberately absent, for the reason in Using Alloy for observability .

Prerequisites: a metrics backend that accepts remote-write and a Loki that accepts pushes, both reachable from inside the cluster (Set up Prometheus and Set up Loki ), plus Helm and kubectl access.

Warning

This DaemonSet is a privileged collector. On the host network it reaches every loopback listener on the node, including the unauthenticated ones (etcd on 2381, KubeGate on 8080, syself-agent on 20257), and it reads root-owned audit files. Keep it in a namespace only your platform team can write to, run no application workloads there, and review its config like a firewall rule.

What the config has to solve#

Four things make this config specific to Syself Autopilot rather than a stock Kubernetes install.

Loopback binding. Most platform metrics endpoints bind to 127.0.0.1, so the agent shares the node's network namespace with hostNetwork: true and dials loopback directly. dnsPolicy: ClusterFirstWithHostNet keeps cluster DNS resolvable from there, which the backend Service names need.
A per-node identity. Every agent dials the same 127.0.0.1:<port>, so without intervention every node reports the same instance label. A remote-write receiver then sees many agents writing one series, interleaves the samples, and rejects the overlap as out of order (err-mimir-sample-out-of-order on Grafana Mimir). The config stamps the node name as instance and node on every target, injected through the Downward API.
Node-role gating. etcd, kube-controller-manager, kube-scheduler, KubeGate, and syself-tunnel-server run only on control planes; syself-proxy only on workers; the CSI node plugin only on cloud workers. Give every node the same target list and the nodes missing a component report connection refused forever, behind a TargetDown alert that never clears. The config reads the local node's labels and keeps each job only where that component actually runs. Control-plane nodes carry node-role.kubernetes.io/control-plane with an empty value, so the gate checks that the label is there rather than what it says; workers carry node.kubernetes.io/worker=true, and cloud nodes carry autopilot.syself.com/machine-type=hcloud. On a cluster with bare-metal workers, the CSI node job needs both labels: a worker-only gate keeps the bare-metal nodes, where that port is never listening.
Self-signed certificates on loopback. kube-controller-manager, kube-scheduler, and the kubelet serve HTTPS with a certificate the agent cannot verify from loopback: self-signed, or with the node IP as its only SAN. Those jobs set insecure_skip_verify. The connection never leaves the node, so there is nothing on the wire to intercept. The API server is the exception: it listens on loopback too and its serving certificate names 127.0.0.1, so that job verifies against the ServiceAccount CA bundle rather than skipping verification.

Deploy it#

Write the values file #

The chart's own RBAC covers everything this agent needs. rbac.create: true grants the discovery permissions plus nodes/metrics and get on the /metrics non-resource URL, which is exactly what the token-authenticated control-plane and kubelet jobs authorize against, so no extra ClusterRole is required.

mounts.varlog: true mounts the host's whole /var/log, which is where the container logs, the journal, and all three audit streams live, so one flag replaces a list of hostPath volumes. All three audit streams are mode 0600, owned by root except the KubeGate log, which belongs to the unprivileged user KubeGate runs as. Either way no unprivileged reader can open them, so the container runs as root.

system-alloy-values.yamlyaml

		controller:
  type: daemonset
  # Reach the node's loopback endpoints, and keep cluster DNS working from there.
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  # Tolerate every taint, so the agent lands on all nodes: control planes,
  # where etcd, the scheduler, the controller-manager, and KubeGate live, and
  # any node your own workloads have tainted. A node without this agent is a
  # node with no telemetry at all.
  tolerations:
    - operator: Exists
  volumes:
    extra:
      # Hubble flow export, if you enabled it. Drop this if you did not.
      - name: hubble
        hostPath: {path: /var/run/cilium/hubble}
 
alloy:
  # Read the 0600 root-owned audit files.
  securityContext:
    runAsUser: 0
    runAsNonRoot: false
  mounts:
    # /var/log covers container logs, the journal, and the audit streams.
    varlog: true
    extra:
      - name: hubble
        mountPath: /var/run/cilium/hubble
        readOnly: true
  extraEnv:
    - name: NODE_NAME
      valueFrom:
        fieldRef: {fieldPath: spec.nodeName}
  # Buffer samples on disk so a backend restart costs latency, not data.
  storagePath: /var/lib/alloy
  configMap:
    content: |
      // ===================== Destinations =====================
 
      prometheus.remote_write "central" {
        endpoint {
          url = "http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/write"
        }
      }
 
      loki.write "central" {
        endpoint {
          url = "http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push"
        }
      }
 
      // ===================== Node identity and role gates =====================
 
      discovery.kubernetes "nodes" {
        role = "node"
      }
 
      // Keep only this node's entry, and stamp its name so every node's series
      // stays distinct at the receiver.
      discovery.relabel "local_node" {
        targets = discovery.kubernetes.nodes.targets
 
        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          target_label  = "instance"
        }
        rule {
          source_labels = ["__meta_kubernetes_node_name"]
          target_label  = "node"
        }
      }
 
      // Yields one target on a control plane and nothing elsewhere. The role
      // label's value is empty here, so test for its presence.
      discovery.relabel "control_plane" {
        targets = discovery.relabel.local_node.output
 
        rule {
          source_labels = ["__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_control_plane"]
          regex         = "true"
          action        = "keep"
        }
      }
 
      // Yields one target on a worker and nothing elsewhere.
      discovery.relabel "worker" {
        targets = discovery.relabel.local_node.output
 
        rule {
          source_labels = ["__meta_kubernetes_node_label_node_kubernetes_io_worker"]
          regex         = "true"
          action        = "keep"
        }
      }
 
      // The CSI node plugin runs on cloud workers only, so its gate needs the
      // machine type as well. A worker-only gate keeps bare-metal nodes too,
      // where port 9189 is never listening.
      discovery.relabel "cloud_worker" {
        targets = discovery.relabel.worker.output
 
        rule {
          source_labels = ["__meta_kubernetes_node_label_autopilot_syself_com_machine_type"]
          regex         = "hcloud"
          action        = "keep"
        }
      }
 
      // ===================== Every node, plain HTTP =====================
 
      prometheus.scrape "node_local" {
        targets = [
          {__address__ = "127.0.0.1:9962",  job = "cilium-agent",        instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:9965",  job = "hubble",              instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:20257", job = "syself-agent",        instance = sys.env("NODE_NAME")},
          {__address__ = "127.0.0.1:8182",  job = "syself-tunnel-agent", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // containerd serves its metrics on /v1/metrics, not the usual /metrics,
      // so it needs its own block rather than a line in the list above.
      prometheus.scrape "containerd" {
        targets = [
          {__address__ = "127.0.0.1:1338", job = "containerd", instance = sys.env("NODE_NAME")},
        ]
        metrics_path = "/v1/metrics"
        forward_to   = [prometheus.remote_write.central.receiver]
      }
 
      // The agent's own metrics. Without this nothing watches the collector,
      // and a collector that stops looks exactly like a quiet cluster.
      prometheus.scrape "alloy_self" {
        targets = [
          {__address__ = "127.0.0.1:12345", job = "alloy-system", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // node-exporter is your install, not the platform's.
      // Delete this block if you skipped it.
      prometheus.scrape "node_exporter" {
        targets = [
          {__address__ = "127.0.0.1:9100", job = "node-exporter", instance = sys.env("NODE_NAME")},
        ]
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // ===================== Every node, HTTPS with a token =====================
 
      prometheus.scrape "kubelet" {
        targets = [
          {__address__ = "127.0.0.1:10250", job = "kubelet", instance = sys.env("NODE_NAME")},
        ]
        scheme           = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // Per-container CPU, memory, and network, from the kubelet's own path.
      prometheus.scrape "cadvisor" {
        targets = [
          {__address__ = "127.0.0.1:10250", job = "cadvisor", instance = sys.env("NODE_NAME")},
        ]
        scheme            = "https"
        metrics_path      = "/metrics/cadvisor"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // ===================== Control planes only =====================
 
      discovery.relabel "etcd" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:2381"
        }
      }
 
      prometheus.scrape "etcd" {
        targets    = discovery.relabel.etcd.output
        job_name   = "etcd"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // The API server listens on loopback as well as the node IP, and its
      // serving certificate names 127.0.0.1, so this is the one control-plane
      // job that verifies TLS properly instead of skipping it.
      discovery.relabel "apiserver" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:6443"
        }
      }
 
      prometheus.scrape "apiserver" {
        targets           = discovery.relabel.apiserver.output
        job_name          = "kube-apiserver"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          ca_file = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "kubegate" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:8080"
        }
      }
 
      prometheus.scrape "kubegate" {
        targets    = discovery.relabel.kubegate.output
        job_name   = "kubegate"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "tunnel_server" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:8181"
        }
      }
 
      prometheus.scrape "tunnel_server" {
        targets    = discovery.relabel.tunnel_server.output
        job_name   = "syself-tunnel-server"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "controller_manager" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:10257"
        }
      }
 
      prometheus.scrape "controller_manager" {
        targets           = discovery.relabel.controller_manager.output
        job_name          = "kube-controller-manager"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "scheduler" {
        targets = discovery.relabel.control_plane.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:10259"
        }
      }
 
      prometheus.scrape "scheduler" {
        targets           = discovery.relabel.scheduler.output
        job_name          = "kube-scheduler"
        scheme            = "https"
        bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        tls_config {
          insecure_skip_verify = true
        }
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // ===================== Workers only =====================
 
      // syself-proxy runs on every worker; the CSI node plugin only on cloud
      // workers, so the two jobs use different gates.
 
      discovery.relabel "csi_node" {
        targets = discovery.relabel.cloud_worker.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9189"
        }
      }
 
      prometheus.scrape "csi_node" {
        targets    = discovery.relabel.csi_node.output
        job_name   = "csi-node"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "syself_proxy" {
        targets = discovery.relabel.worker.output
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9587"
        }
      }
 
      prometheus.scrape "syself_proxy" {
        targets    = discovery.relabel.syself_proxy.output
        job_name   = "syself-proxy"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // ===================== Addons on the pod network =====================
 
      // The Cilium operator, the CCM, hubble-relay, the CSI controller, and
      // CoreDNS run as ordinary pods. Each agent scrapes only the ones on its
      // own node, so the work spreads without any target being scraped twice.
      discovery.kubernetes "addon_pods" {
        role = "pod"
        namespaces {
          names = ["kube-system"]
        }
      }
 
      discovery.relabel "addons" {
        targets = discovery.kubernetes.addon_pods.targets
 
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "node"
        }
      }
 
      // Cilium operator binds its metrics to loopback like the agent does.
      discovery.relabel "cilium_operator" {
        targets = discovery.relabel.addons.output
 
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          regex         = "cilium-operator"
          action        = "keep"
        }
        // These pods declare more than one port, and pod discovery emits a
        // target for each. Keep the metrics port so the endpoint is scraped
        // once, not once per port.
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "prometheus"
          action        = "keep"
        }
        rule {
          target_label = "__address__"
          replacement  = "127.0.0.1:9963"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "instance"
        }
      }
 
      prometheus.scrape "cilium_operator" {
        targets    = discovery.relabel.cilium_operator.output
        job_name   = "cilium-operator"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // The rest serve on the pod network, so scrape the pod IP.
      discovery.relabel "ccm" {
        targets = discovery.relabel.addons.output
 
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          regex         = "ccm"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:8233"
          target_label  = "__address__"
        }
      }
 
      prometheus.scrape "ccm" {
        targets    = discovery.relabel.ccm.output
        job_name   = "ccm"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "hubble_relay" {
        targets = discovery.relabel.addons.output
 
        rule {
          source_labels = ["__meta_kubernetes_pod_label_k8s_app"]
          regex         = "hubble-relay"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "prometheus"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:9966"
          target_label  = "__address__"
        }
      }
 
      prometheus.scrape "hubble_relay" {
        targets    = discovery.relabel.hubble_relay.output
        job_name   = "hubble-relay"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.relabel "csi_controller" {
        targets = discovery.relabel.addons.output
 
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_label_app_kubernetes_io_component"]
          separator     = ";"
          regex         = "csi;controller"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_port_name"]
          regex         = "metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__address__"]
          regex         = "([^:]+)(?::\\d+)?"
          replacement   = "$1:9189"
          target_label  = "__address__"
        }
      }
 
      prometheus.scrape "csi_controller" {
        targets    = discovery.relabel.csi_controller.output
        job_name   = "csi-controller"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      discovery.kubernetes "endpoints" {
        role = "endpoints"
      }
 
      discovery.relabel "coredns" {
        targets = discovery.kubernetes.endpoints.targets
 
        rule {
          source_labels = ["__meta_kubernetes_service_name", "__meta_kubernetes_endpoint_port_name"]
          separator     = ";"
          regex         = "kube-dns;metrics"
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_endpoint_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
      }
 
      prometheus.scrape "coredns" {
        targets    = discovery.relabel.coredns.output
        job_name   = "coredns"
        forward_to = [prometheus.remote_write.central.receiver]
      }
 
      // ===================== Container logs =====================
 
      discovery.kubernetes "pods" {
        role = "pod"
      }
 
      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pods.targets
 
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          regex         = sys.env("NODE_NAME")
          action        = "keep"
        }
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_node_name"]
          target_label  = "node"
        }
        // /var/log/pods/<namespace>_<pod>_<uid>/<container>/*.log
        rule {
          source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
          separator     = "/"
          replacement   = "/var/log/pods/*$1/*.log"
          target_label  = "__path__"
        }
      }
 
      // discovery.relabel produced a glob per container; local.file_match is
      // what expands it into the real files. loki.source.file does not expand
      // globs itself, so skipping this step tails nothing at all.
      local.file_match "pod_logs" {
        path_targets = discovery.relabel.pod_logs.output
      }
 
      loki.source.file "pod_logs" {
        targets    = local.file_match.pod_logs.targets
        forward_to = [loki.process.pod_logs.receiver]
      }
 
      loki.process "pod_logs" {
        // Unwrap the CRI envelope so the message, not the wrapper, is the line.
        stage.cri {}
        forward_to = [loki.write.central.receiver]
      }
 
      // ===================== journald =====================
 
      discovery.relabel "journal" {
        targets = []
 
        // Keep the units worth reading. Everything else is volume you pay for
        // and never query.
        rule {
          source_labels = ["__journal__systemd_unit"]
          regex         = "(kubelet|containerd|syself-.*)\\.service"
          action        = "keep"
        }
        rule {
          source_labels = ["__journal__systemd_unit"]
          target_label  = "unit"
        }
      }
 
      loki.source.journal "host" {
        path          = "/var/log/journal"
        relabel_rules = discovery.relabel.journal.rules
        labels        = {job = "journald", node = sys.env("NODE_NAME")}
        forward_to    = [loki.write.central.receiver]
      }
 
      // ===================== Audit streams =====================
 
      // The Linux audit daemon, on every node.
      local.file_match "auditd" {
        path_targets = [{
          __path__ = "/var/log/audit/audit.log",
          job      = "auditd",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }
 
      loki.source.file "auditd" {
        targets    = local.file_match.auditd.targets
        forward_to = [loki.write.central.receiver]
      }
 
      // The Kubernetes API and KubeGate audit trails, control planes only. On a
      // worker these paths are absent or empty, so nothing is tailed and the
      // agent stays quiet.
      local.file_match "api_audit" {
        path_targets = [{
          __path__ = "/var/log/kube-apiserver/kube-apiserver.log",
          job      = "kube-apiserver-audit",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }
 
      loki.source.file "api_audit" {
        targets    = local.file_match.api_audit.targets
        forward_to = [loki.process.api_audit.receiver]
      }
 
      // Lift the fields you query on out of the JSON envelope into structured metadata.
      loki.process "api_audit" {
        stage.json {
          expressions = {verb = "verb", user = "user.username", uri = "requestURI"}
        }
        stage.structured_metadata {
          values = {verb = "verb", user = "user", uri = "uri"}
        }
        forward_to = [loki.write.central.receiver]
      }
 
      local.file_match "kubegate_audit" {
        path_targets = [{
          __path__ = "/var/log/kubegate/audit.log",
          job      = "kubegate-audit",
          log_type = "audit",
          node     = sys.env("NODE_NAME"),
        }]
      }
 
      loki.source.file "kubegate_audit" {
        targets    = local.file_match.kubegate_audit.targets
        forward_to = [loki.write.central.receiver]
      }

Install the chart #

		$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm upgrade -i system-alloy grafana/alloy \
  --namespace monitoring --create-namespace \
  --version <chart-version> \
  --values system-alloy-values.yaml

Pin --version to a chart release you have tested, so an upgrade does not change collection behavior underneath you.

Confirm a pod runs on every node #

		$ kubectl -n monitoring get pods -l app.kubernetes.io/instance=system-alloy -o wide

Expect one pod per node, control planes included. A control-plane node with no pod means the toleration did not apply.

Check the targets #

Alloy serves a UI that lists every component and its targets, which is the fastest way to see what a single node is actually collecting:

		$ kubectl -n monitoring port-forward daemonset/system-alloy 12345:12345

Open http://localhost:12345 and look at the prometheus.scrape components. On a control-plane pod the etcd, kubegate, controller_manager, scheduler, and tunnel_server components each show one target up, and csi_node and syself_proxy show none. On a worker pod that reverses. A component with no targets on the node where the component does not belong is the gating working, not a fault.

Then confirm the samples arrive at the backend, one series per node rather than one shared series:

text

		count(up{job="etcd"}) by (instance)
count(up{job="cilium-agent"}) by (instance)

Add the Hubble flow log#

If you turned on Hubble flow export (Alert on dropped packets ), the verdict log is a file on the node like any other, and the values file above already mounts its directory. Add a source for it:

terraform

		local.file_match "hubble_flows" {
  path_targets = [{
    __path__ = "/var/run/cilium/hubble/events.log",
    job      = "hubble-flows",
    node     = sys.env("NODE_NAME"),
  }]
}
 
loki.source.file "hubble_flows" {
  targets    = local.file_match.hubble_flows.targets
  forward_to = [loki.write.central.receiver]
}

Keep the log volume down#

The journald rule above keeps kubelet, containerd, and the syself-* units and drops the rest, which is most of the volume for none of the value. Container logs are the other large contributor and vary entirely with your workloads, so drop or sample the high-volume namespaces rather than paying to store what you will not query. When Loki is slow or an incident floods logs, Alloy's disk buffer at storagePath absorbs the burst; size that volume so a spike degrades into delay instead of taking the node's memory with it. Log retention and sizing has the per-node estimates to size against.

With the node signals flowing, add the Application Alloy for your workloads' metrics and traces. For what the series you just started collecting actually tell you, see Control-plane metrics and Data-plane and cluster component metrics . If a target will not come up, Troubleshoot the collectors covers the failures worth knowing by name.

Using Alloy for observability

Run one Grafana Alloy DaemonSet for infrastructure telemetry and one clustered Alloy StatefulSet for application telemetry, so a runaway application cannot take your node telemetry down with it.

Deploy the Application Alloy

Run a clustered Grafana Alloy StatefulSet that scrapes your workloads through their existing ServiceMonitors and receives OTLP traces, sharded across replicas.