Set up Loki
On this page you install Loki as the store for your logs. Loki only stores and answers queries, the same role Prometheus plays for metrics. The System Alloy is what ships logs into Loki from every node. Back it with object storage (an S3-compatible bucket) so the data survives node replacement, and add it to Grafana so you can query logs next to your metrics.
Install and connect Loki
Install Loki
This chart installs Loki without a log collector.
$ helm repo add grafana https://grafana.github.io/helm-charts
# Set loki.auth_enabled=true to require tenant headers for multi-tenancy
$ helm install loki grafana/loki \
--namespace monitoring --create-namespace \
--set deploymentMode=SingleBinary \
--set loki.auth_enabled=false \
--set loki.commonConfig.replication_factor=1 \
--set singleBinary.replicas=1 \
--set loki.storage.type=filesystem \
--set read.replicas=0 \
--set backend.replicas=0 \
--set write.replicas=0 \
--set loki.useTestSchema=true \
--set gateway.nginxConfig.enableIPv6=false \
--set chunksCache.enabled=false \
--set resultsCache.enabled=false \
--set loki.limits_config.max_line_size=2MB
The flags above are a single-binary starting point with filesystem storage, which is enough to start logs flowing and query them. Replace loki.useTestSchema and the filesystem backend with a real schema and object storage before you keep anything you care about, as the next section covers.
Important
max_line_size is raised from Loki's 256 KB default because a single Kubernetes API audit event can be larger than that, and Loki rejects an oversized entry outright instead of truncating it. Your audit trail then quietly loses its largest events while every ordinary one arrives. Watch loki_write_dropped_entries_total{reason="ingester_error"} on the collector to catch it, and raise the limit further if your audit policy captures large request bodies.
The deployment mode is a scale choice:
Everything runs in one process. This is simple and enough for one cluster's logs. Start here.
The parts split into separate deployments you scale independently: ingesters that receive incoming logs, queriers that answer searches, and a compactor that merges stored logs and applies retention. This is the shape for a fleet's worth of logs or a multi-tenant store. Move to it when ingestion or query load outgrows one process.
Back it with object storage
Loki's chunks (its stored log data) belong on object storage, not on a local disk that a node replacement takes with it. Point it at a native bucket or JuiceFS over object storage , and set a retention period. Object storage is what makes long retention cheap and durable, the same reason it backs the long-term metric stores.
Note
The single-binary Loki claims a PersistentVolumeClaim from the default standard StorageClass, and standard volumes provision only on cloud nodes. On a cluster that also has bare-metal nodes, Loki can get scheduled onto one of them, and its claim then stays Pending. Pin it to a cloud node with a nodeSelector on autopilot.syself.com/machine-type: hcloud, or set storageClassName to a local class through TopoLVM . See the storage model for how each class behaves through node replacement.
Add it to Grafana
Add Loki as a data source (http://loki-gateway.monitoring.svc.cluster.local) so you query logs beside metrics. A spike on a metrics panel and the log lines behind it are then one click apart in the same Grafana.
Multi-tenancy for many clients
Loki's tenant model (the X-Scope-OrgID header) keeps each tenant's logs separate in one Loki. For an agency, give each client cluster its own tenant id so one client can never query another's logs, and route queries per tenant behind SSO. See Multi-cluster observability .
Size the ingesters and compactor
Size the ingesters for your peak ingest rate and the compactor for your retention and query load. A rough start: estimate per-node ingest (Log retention and sizing ), multiply by node count, and give ingesters headroom for the burst during an incident when log volume jumps. Running this on your own object storage keeps the log store on capacity you already own.
With the store up, start the log lines shipping: Deploy the System Alloy points its loki.write at the gateway URL above and tails every source on the node.
Log sources on a sealed node
A sealed Syself node has a read-only root and a writable /var, and this page lists every log stream that lives there and disappears on reprovision.
Collect container and journald logs
Container stdout and the systemd journal are the two operational log streams on every node, and this page covers what each holds, how they are labelled, and how to cut the volume.