Collect logs with the log collector

When a node itself is the problem, events and status are not enough: support needs the node's logs. You do not have to hunt for them. Every node ships a collector that pulls the logs from the Kubernetes components, the system services, and the operating system into one compressed file. Run it on the affected node, get the bundle off, and support has the full picture.

Two ways to get the bundle off the node

You run the collector on the node, then copy the bundle to your machine. There are two ways, depending on whether you have SSH access:

• With SSH (key-only, port 100): run the collector over SSH, then copy the bundle out with scp.
• Without SSH (kubectl only): run the collector as a short-lived privileged Job and stream the bundle out through kubectl logs. This works whenever the API server is reachable, even when SSH to the node is not.

Option A: over SSH

SSH into the node and run the collector:

				$ /opt/log-collector.sh
			

The collector prints the bundle it wrote, with a name generated for that run:

				/var/log/syself_<node-name>_<timestamp>_<collector-version>.tar.gz
			

Nothing produces this bundle on a schedule. Run the collector first, then copy the file. Copying without running the collector either fails with "No such file or directory" or hands you a stale bundle from whenever someone last ran it.

Then, from a terminal on your local machine (not the node), copy the file with scp on port 100. Replace <path-to-ssh-key> with the path to your SSH key, <node-ip> with the node's IP address, and <bundle> with the file name the collector printed:

				$ scp -P 100 -i <path-to-ssh-key> root@<node-ip>:/var/log/<bundle> .
			

The file lands in your current directory.

Option B: with kubectl, no SSH

When you cannot SSH to the node but the API server is reachable, run the collector as a privileged Job and pull the bundle back through kubectl logs. The Job runs the collector, then writes the tarball to its stdout as base64, wrapped in BEGIN/END markers.

				apiVersion: batch/v1
kind: Job
metadata:
  name: log-collector
  namespace: kube-system
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      hostPID: true
      nodeName: <node-name> # the node you want the bundle from
      containers:
        - name: collect
          image: busybox
          securityContext:
            privileged: true
          command:
            - sh
            - -c
            - |
              chroot /host /opt/log-collector.sh >/dev/null 2>&1
              echo "BEGIN SYSELF BUNDLE"
              base64 "$(ls -t /host/var/log/syself_*.tar.gz | head -1)"
              echo "END SYSELF BUNDLE"
          volumeMounts:
            - name: host
              mountPath: /host
      volumes:
        - name: host
          hostPath:
            path: /
			

Apply it, wait for it to finish, then decode the bundle out of the logs:

				$ kubectl apply -f log-collector-job.yaml
$ kubectl -n kube-system wait --for=condition=complete job/log-collector --timeout=120s
$ kubectl -n kube-system logs job/log-collector \
  | sed -n '/BEGIN SYSELF BUNDLE/,/END SYSELF BUNDLE/p' | sed '1d;$d' \
  | base64 -d > report.tar.gz
$ kubectl -n kube-system delete job/log-collector
			

kubectl cp does not work here: it cannot read from a finished pod. Use the kubectl logs path above.

Attach the bundle to a support case

Attach the bundle when you contact support, along with the cluster name, the affected node name, and a description of the problem. See How do I get support? .

If the affected node is a bare-metal server and you cannot SSH into it, give support direct access instead. See Grant Syself bare-metal access .

Related

• Where to start
• Grant Syself bare-metal access
• Emergency SSH access