Collect logs with the log collector

When a node itself is the problem, events and status are not enough: support needs the node's logs. You do not have to hunt for them. Every node ships a collector that pulls the logs from the Kubernetes components, the system services, and the operating system into one compressed file. Run it on the affected node, get the bundle off, and support has the full picture.

Two ways to get the bundle off the node#

You run the collector on the node, then copy the bundle to your machine. There are two ways, depending on whether you have SSH access:

• With SSH (key-only, port 100): run the collector over SSH, then copy the bundle out with scp.
• Without SSH (kubectl only): run the collector as a short-lived privileged Job and stream the bundle out through kubectl logs. This works whenever the API server is reachable, even when SSH to the node is not.

Option A: over SSH#

SSH into the node and run the collector:

		$ syself-log-collector
	

The collector writes the bundle to:

		/var/log/syself-node-report/latest.tar.gz
	

Nothing produces this bundle on a schedule. Run the collector first, then copy the file. Copying without running the collector either fails with "No such file or directory" or hands you a stale bundle from whenever someone last ran it.

Then, from a terminal on your local machine (not the node), copy the file with scp on port 100. Replace <path-to-ssh-key> with the path to your SSH key and <node-ip> with the node's IP address:

		$ scp -P 100 -i <path-to-ssh-key> root@<node-ip>:/var/log/syself-node-report/latest.tar.gz .
	

The file lands in your current directory.

Option B: with kubectl, no SSH#

When you cannot SSH to the node but the API server is reachable, run the collector as a privileged Job and pull the bundle back through kubectl logs. The collector writes the tarball to its stdout as base64, wrapped in BEGIN/END markers.

		apiVersion: batch/v1
kind: Job
metadata:
  name: log-collector
  namespace: kube-system
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      hostPID: true
      nodeName: <node-name> # the node you want the bundle from
      containers:
        - name: collect
          image: busybox
          securityContext:
            privileged: true
          command:
            [
              "chroot",
              "/host",
              "/usr/bin/syself-log-collector",
              "--stdout",
              "--stdout-encoding=base64"
            ]
          volumeMounts:
            - name: host
              mountPath: /host
      volumes:
        - name: host
          hostPath:
            path: /
	

Apply it, wait for it to finish, then decode the bundle out of the logs:

		$ kubectl apply -f log-collector-job.yaml
$ kubectl -n kube-system wait --for=condition=complete job/log-collector --timeout=120s
$ kubectl -n kube-system logs job/log-collector \
  | sed -n '/BEGIN SYSELF BUNDLE/,/END SYSELF BUNDLE/p' | sed '1d;$d' \
  | base64 -d > report.tar.gz
$ kubectl -n kube-system delete job/log-collector
	

kubectl cp does not work here: it cannot read from a finished pod. Use the kubectl logs path above.

What the bundle includes#

On a control-plane node, the collector's audit component bundles the Kubernetes API server audit directory. The host auditd log lives at /var/log/audit/ and you read it directly, since the collector does not fold it into the bundle. See Retrieve audit logs .

Attach the bundle to a support case#

Attach the bundle when you contact support, along with the cluster name, the affected node name, and a description of the problem. See Get support .

If the affected node is a bare-metal server and you cannot SSH into it, give support direct access instead. See Grant Syself bare-metal access .

Related#

• Get support
• What to collect first
• Grant Syself bare-metal access
• Emergency SSH access