Set up Loki
Install Loki to store and query logs beside your metrics in Grafana: same query surface, same dashboards, with logs alongside metrics. Back it with object storage (bucket storage like S3) so it scales and survives node replacement, and add it as a Grafana data source. It is the store only; the System Alloy is what ships logs into it from every node.
Install Loki#
This installs the store. No collector comes with it, and none is needed: the agent that already scrapes each node's metrics tails its logs too, so there is no second DaemonSet to run or upgrade.
$ helm repo add grafana https://grafana.github.io/helm-charts
# Set loki.auth_enabled=true to require tenant headers for multi-tenancy
$ helm install loki grafana/loki \
--namespace monitoring --create-namespace \
--set deploymentMode=SingleBinary \
--set loki.auth_enabled=false \
--set loki.commonConfig.replication_factor=1 \
--set singleBinary.replicas=1 \
--set loki.storage.type=filesystem \
--set read.replicas=0 \
--set backend.replicas=0 \
--set write.replicas=0 \
--set loki.useTestSchema=true \
--set gateway.nginxConfig.enableIPv6=false \
--set chunksCache.enabled=false \
--set resultsCache.enabled=false \
--set loki.limits_config.max_line_size=2MB
The flags above are a single-binary starting point with filesystem storage, which is enough to start logs flowing and query them. Replace loki.useTestSchema and the filesystem backend with a real schema and object storage before you keep anything you care about, as the next section covers.
Important
max_line_size is raised from Loki's 256 KB default because a single Kubernetes API audit event can be far larger than that. An audit event that captures a request body, a CRD apply for instance, runs to several hundred kilobytes, and Loki rejects any entry over the limit outright rather than truncating it. The collector logs the rejection and drops the line:
server returned HTTP status 400 Bad Request (400): max entry size '262144' bytes exceeded
for stream '{job="kube-apiserver-audit", ...}' while adding an entry with length '609745' bytes
The effect is that the biggest and often most interesting audit events go missing while every ordinary one arrives, so the trail looks complete and is not. Watch loki_write_dropped_entries_total{reason="ingester_error"} on the collector to catch it. Raise the limit further if your audit policy captures large request bodies.
Single-binary or distributed#
The deployment mode is a scale choice:
Everything runs in one process. This is simple and enough for one cluster's logs. Start here.
The parts split into separate deployments you scale independently: ingesters that receive incoming logs, queriers that answer searches, and a compactor that merges stored logs and applies retention. This is the shape for a fleet's worth of logs or a multi-tenant store. Move to it when ingestion or query load outgrows one process.
Back it with object storage#
Loki's chunks (its stored log data) belong on object storage, not on a local disk that a node replacement takes with it. Point it at a native bucket or JuiceFS over object storage , and set a retention period. Object storage is what makes long retention cheap and durable, the same reason it backs the long-term metric stores.
Note
The single-binary Loki claims a PersistentVolumeClaim from the default standard StorageClass, which is backed by Hetzner Cloud volumes and provisions only on cloud nodes. On a cluster with bare-metal nodes, the install command above sets no placement, so if Loki is scheduled onto a bare-metal node the standard claim cannot be provisioned there and the pod stays Pending. Pin Loki to a cloud node with a nodeSelector (autopilot.syself.com/machine-type: hcloud), or set storageClassName to a local class (local-nvme, also local-ssd and local-hdd) that binds to that node's local disk through TopoLVM , which you install. The nodeSelector is only needed for the standard path: a local class places itself, because TopoLVM provisions on a prepared bare-metal server and the bound volume pins the pod to it. A local volume is also tied to one node and does not survive its replacement, so back Loki with object storage (above) for anything you keep. See the storage model for how each class behaves through node replacement.
Add it to Grafana#
Add Loki as a data source (http://loki-gateway.monitoring.svc.cluster.local) so you query logs beside metrics. A spike on a metrics panel and the log lines behind it are then one click apart in the same Grafana.
Multi-tenancy for many clients#
Loki's tenant model (the X-Scope-OrgID header) keeps each tenant's logs separate in one Loki. For an agency, give each client cluster its own tenant id so one client can never query another's logs, and route queries per tenant behind SSO. See Multi-cluster observability .
Size the ingesters and compactor#
Size the ingesters for your peak ingest rate and the compactor for your retention and query load. A rough start: estimate per-node ingest (Log retention and sizing ), multiply by node count, and give ingesters headroom for the burst during an incident when log volume jumps. Running this on your own object storage keeps the log store on capacity you already own.
With the store up, start the log lines shipping: Deploy the System Alloy points its loki.write at the gateway URL above and tails every source on the node.
Log sources on a sealed node
A sealed Syself node has a read-only root and a writable /var, and this page lists every log stream that lives there and disappears on reprovision.
Collect container and journald logs
Container stdout and the systemd journal are the two operational log streams on every node, and this page covers what each holds, how they are labelled, and how to cut the volume.