Skip to main content

Troubleshoot an unreachable machine

Inspect 1.36

When a provisioned server fails to join the Kubernetes cluster and becomes completely unreachable over the network, you cannot access it through SSH or kubectl.

In this state, the standard syself-log-collector cannot run. To diagnose the failure on the machine, you must use out-of-band tools: a remote console to observe early-boot issues, and the Hetzner rescue system to mount the disk and extract logs.

Step 1: Pause automation to prevent machine deletion

When a machine fails to register with Kubernetes before its timeout expires, Cluster API and the Hetzner provider controllers mark it as failed and prepare to delete or reprovision it. Reprovisioning wipes the root disk, destroying the diagnostic logs.

Freeze controller actions on the machine object before you begin troubleshooting:

		$ kubectl -n <cluster-namespace> annotate machine <machine-name> cluster.x-k8s.io/paused="true"

If troubleshooting a bare-metal server, you can alternatively set spec.maintenanceMode: true on the corresponding HetznerBareMetalHost resource. For more details on pausing automation, see .

Step 2: Confirm network unreachability

Verify whether the server has completely lost network connectivity or is only rejecting specific services:

  1. Test basic network reachability with ICMP: 
    		$ ping <server-ip>
  2. Test SSH connectivity on both the Syself default port (100): 
    		$ ssh -v -p 100 -i <path-to-key> root@<server-ip>

If ping fails or SSH connections time out with no response, the machine either crashed during boot or its network interface failed to initialize.

Step 3: Inspect early boot with a remote console

Early boot issues—such as BIOS configuration problems, kernel panics, device-mapper verity (dm-verity) hash mismatches, or missing NIC drivers—occur before systemd starts and are never written to disk logs. A remote console is the only way to observe them.

Hetzner Cloud servers feature a built-in browser VNC console that connects directly to the virtual machine display:

  1. Open the Hetzner Cloud Console⁠ and navigate to your project.
  2. Select the affected server and click the Console button in the upper right. See Hetzner Cloud Console documentation⁠.
  3. From the server actions menu, trigger a reboot (Power -> Reboot or Reset).
  4. Watch the console display in real time as the virtual machine boots.

What to watch for during boot

Pay close attention to error messages on the console screen:

  • dm-verity errors: device-mapper: verity: data block ... is corrupted indicates an integrity failure in the root filesystem.
  • Kernel panics: Call traces identifying missing drivers, memory faults, or CPU panics.
  • Network initialization: Messages regarding link status or interface naming (e.g., virtio_net on cloud, or hardware drivers such as bnxt_en, e1000e, igb, or r8169 on bare metal).
  • Systemd freezes: Services hanging indefinitely during first-boot initialization.

Take screenshots of any errors or panic traces displayed on the console viewer to attach to your support case.

Step 4: Boot into the Hetzner rescue system

If the server boots without a kernel panic but remains inaccessible over the network, boot into Hetzner’s RAM-based rescue environment to access the local disk files directly.

  1. In the Hetzner Cloud Console⁠, select the server and navigate to the Rescue tab.
  2. Choose your public SSH key, and click Enable Rescue & Power Cycle. See Hetzner Cloud Rescue documentation⁠.
  3. The virtual machine reboots into the rescue environment.

Once the server has rebooted into rescue mode, connect via SSH:

		$ ssh root@<server-ip> -i <path to ssh key>

Step 5: Mount the disk and inspect the logs

Both Hetzner Cloud and bare-metal servers running Syself Linux use standard partition labels. Inspect the partitions and mount the machine’s data filesystem.

  1. List available storage devices and partitions:

    		# lsblk -f

    Identify the partition labeled DATA (formatted as ext4, typically /dev/vda5 on Hetzner Cloud or /dev/nvme0n1p5 / /dev/sda5 on bare metal).

  2. Mount the partition read-only under /mnt:

    		# mount -o ro /dev/disk/by-label/DATA /mnt

  3. Review the provisioner debug logs:

    		# ls -la /mnt/var/log/syself/provision/

    This directory contains diagnostic artifacts collected by the Syself provisioner before rebooting, including partition dumps, storage layout, sysctl parameters, and image verification data.

  4. Check the first-boot activation log:

    		# cat /mnt/var/log/syself-activate.log

  5. (Optional) Check offline systemd journal logs:

    		# journalctl -D /mnt/var/log/journal
Important

The existence of /var/log/syself-activate.log is critical for diagnosis:

  • Log does not exist: The server failed during early boot (bootloader, kernel panic, or dm-verity verification) before the operating system reached the activation service.
  • Log exists: Inspect the end of the file. It records network configuration steps, certificate generation, and attempts to join the Kubernetes control plane. Any network failure or token rejection will be logged here.

Step 6: Collect the bundle and contact support

Create an archive containing the diagnostic logs while still in the rescue shell:

		# tar -czf /tmp/unreachable-machine-logs.tar.gz -C /mnt/var/log syself syself-activate.log

From your local machine, copy the archive using scp:

		$ scp root@<server-ip>:/tmp/unreachable-machine-logs.tar.gz .

When submitting a ticket, include:

  • The unreachable-machine-logs.tar.gz archive.
  • Any screenshots captured from the Cloud or KVM console.
  • Output from kubectl describe machine <machine-name>.
  • Server type and identifier (Hetzner Cloud Server ID or Robot Server ID).

For bare-metal servers, if Syself engineers request access to investigate the host directly in Hetzner Robot, follow .

Step 7: Cleanup and next steps

  1. Unmount the disk before rebooting: 
    		# umount /mnt
  2. If the investigation concluded that the OS image was corrupted or failed verification, the machine must be reprovisioned. See .
  3. Return the server to regular boot:
    • Hetzner Cloud: In the Cloud Console, ensure rescue mode is disabled, then power cycle the server.
    • Bare Metal: In Hetzner Robot, reset the server without activating rescue mode.
  4. Once ready for automated management to resume, remove the pause annotation: 
    		$ kubectl -n <cluster-namespace> annotate machine <machine-name> cluster.x-k8s.io/paused-

Related

Grant Syself bare-metal access

Put a Hetzner bare-metal server into maintenanceMode, create a temporary Hetzner Robot admin login for Syself support, then revoke it and let the server rejoin the cluster.

FAQ

Answers to the most common questions about Syself Autopilot: what it is, how billing works, and how upgrades work.