When a provisioned server fails to join the Kubernetes cluster and becomes completely unreachable over the network, you cannot access it through SSH or `kubectl`.

In this state, the standard `syself-log-collector` cannot run. To diagnose the failure on the machine, you must use out-of-band tools: a **remote console** to observe early-boot issues, and the **Hetzner rescue system** to mount the disk and extract logs.

## Step 1: Pause automation to prevent machine deletion

When a machine fails to register with Kubernetes before its timeout expires, Cluster API and the Hetzner provider controllers mark it as failed and prepare to delete or reprovision it. Reprovisioning wipes the root disk, destroying the diagnostic logs.

Freeze controller actions on the machine object before you begin troubleshooting:

```console
$ kubectl -n <cluster-namespace> annotate machine <machine-name> cluster.x-k8s.io/paused="true"
```

If troubleshooting a bare-metal server, you can alternatively set `spec.maintenanceMode: true` on the corresponding `HetznerBareMetalHost` resource. For more details on pausing automation, see [Where to start troubleshooting](/docs/hetzner/apalla/support/where-to-start#pause-automation-while-you-investigate).

## Step 2: Confirm network unreachability

Verify whether the server has completely lost network connectivity or is only rejecting specific services:

1. Test basic network reachability with ICMP:
   ```console
   $ ping <server-ip>
   ```
2. Test SSH connectivity on both the Syself default port (`100`):
   ```console
   $ ssh -v -p 100 -i <path-to-key> root@<server-ip>
   ```

If ping fails or SSH connections time out with no response, the machine either crashed during boot or its network interface failed to initialize.

## Step 3: Inspect early boot with a remote console

Early boot issues—such as BIOS configuration problems, kernel panics, device-mapper verity (dm-verity) hash mismatches, or missing NIC drivers—occur before systemd starts and are never written to disk logs. A remote console is the only way to observe them.

<Tabs id="server-platform" items={['Hetzner Cloud', 'Bare Metal']}>

<TabItem value="Hetzner Cloud">

Hetzner Cloud servers feature a built-in browser VNC console that connects directly to the virtual machine display:

1. Open the [Hetzner Cloud Console](https://console.hetzner.cloud/) and navigate to your project.
2. Select the affected server and click the **Console** button in the upper right. See [Hetzner Cloud Console documentation](https://docs.hetzner.com/cloud/servers/getting-started/vnc-console).
3. From the server actions menu, trigger a reboot (**Power** -> **Reboot** or **Reset**).
4. Watch the console display in real time as the virtual machine boots.

</TabItem>

<TabItem value="Bare Metal">

Dedicated servers require a remote KVM console to view the display output:

1. Log in to [Hetzner Robot](https://robot.hetzner.com/) and select the affected server.
2. Order a remote KVM console. Hetzner provides KVM consoles free of charge for up to three hours. Follow [Hetzner’s KVM console documentation](https://docs.hetzner.com/robot/dedicated-server/maintenance/kvm-console/) to request and open the remote viewer.
3. In Hetzner Robot, navigate to the **Reset** tab and issue an **Execute automatic hardware reset** (or send CTRL+ALT+DEL).
4. Watch the console screen as the physical server restarts.

</TabItem>

</Tabs>

### What to watch for during boot

Pay close attention to error messages on the console screen:

- **dm-verity errors:** `device-mapper: verity: data block ... is corrupted` indicates an integrity failure in the root filesystem.
- **Kernel panics:** Call traces identifying missing drivers, memory faults, or CPU panics.
- **Network initialization:** Messages regarding link status or interface naming (e.g., `virtio_net` on cloud, or hardware drivers such as `bnxt_en`, `e1000e`, `igb`, or `r8169` on bare metal).
- **Systemd freezes:** Services hanging indefinitely during first-boot initialization.

Take screenshots of any errors or panic traces displayed on the console viewer to attach to your support case.

## Step 4: Boot into the Hetzner rescue system

If the server boots without a kernel panic but remains inaccessible over the network, boot into Hetzner’s RAM-based rescue environment to access the local disk files directly.

<Tabs id="server-platform" items={['Hetzner Cloud', 'Bare Metal']}>

<TabItem value="Hetzner Cloud">

1. In the [Hetzner Cloud Console](https://console.hetzner.cloud/), select the server and navigate to the **Rescue** tab.
2. Choose your public SSH key, and click **Enable Rescue & Power Cycle**. See [Hetzner Cloud Rescue documentation](https://docs.hetzner.com/cloud/servers/getting-started/rescue-system).
3. The virtual machine reboots into the rescue environment.

</TabItem>

<TabItem value="Bare Metal">

1. In [Hetzner Robot](https://robot.hetzner.com/), select the server and open the **Rescue** tab.
2. Choose your public SSH key, and click **Activate rescue system**. See [Hetzner Robot Rescue documentation](https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system/).
3. Navigate to the **Reset** tab and trigger an **Execute automatic hardware reset**.
4. Wait a few minutes for the server to boot into RAM.

</TabItem>

</Tabs>

Once the server has rebooted into rescue mode, connect via SSH:

```console
$ ssh root@<server-ip> -i <path to ssh key>
```

## Step 5: Mount the disk and inspect the logs

Both Hetzner Cloud and bare-metal servers running Syself Linux use standard partition labels. Inspect the partitions and mount the machine’s data filesystem.

1. List available storage devices and partitions:

   ```console
   # lsblk -f
   ```

   Identify the partition labeled `DATA` (formatted as `ext4`, typically `/dev/vda5` on Hetzner Cloud or `/dev/nvme0n1p5` / `/dev/sda5` on bare metal).

2. Mount the partition read-only under `/mnt`:

   ```console
   # mount -o ro /dev/disk/by-label/DATA /mnt
   ```

3. Review the provisioner debug logs:

   ```console
   # ls -la /mnt/var/log/syself/provision/
   ```

   This directory contains diagnostic artifacts collected by the Syself provisioner before rebooting, including partition dumps, storage layout, sysctl parameters, and image verification data.

4. Check the first-boot activation log:

   ```console
   # cat /mnt/var/log/syself-activate.log
   ```

5. (Optional) Check offline systemd journal logs:
   ```console
   # journalctl -D /mnt/var/log/journal
   ```

> [!IMPORTANT]
> The existence of `/var/log/syself-activate.log` is critical for diagnosis:
>
> - **Log does not exist:** The server failed during early boot (bootloader, kernel panic, or dm-verity verification) before the operating system reached the activation service.
> - **Log exists:** Inspect the end of the file. It records network configuration steps, certificate generation, and attempts to join the Kubernetes control plane. Any network failure or token rejection will be logged here.

## Step 6: Collect the bundle and contact support

Create an archive containing the diagnostic logs while still in the rescue shell:

```console
# tar -czf /tmp/unreachable-machine-logs.tar.gz -C /mnt/var/log syself syself-activate.log
```

From your local machine, copy the archive using `scp`:

```console
$ scp root@<server-ip>:/tmp/unreachable-machine-logs.tar.gz .
```

When submitting a ticket, include:

- The `unreachable-machine-logs.tar.gz` archive.
- Any screenshots captured from the Cloud or KVM console.
- Output from `kubectl describe machine <machine-name>`.
- Server type and identifier (Hetzner Cloud Server ID or Robot Server ID).

For bare-metal servers, if Syself engineers request access to investigate the host directly in Hetzner Robot, follow [Grant Syself bare-metal access](/docs/hetzner/apalla/support/grant-syself-baremetal-access).

## Step 7: Cleanup and next steps

1. Unmount the disk before rebooting:
   ```console
   # umount /mnt
   ```
2. If the investigation concluded that the OS image was corrupted or failed verification, the machine must be reprovisioned. See [Serial console and rescue system](/docs/hetzner/apalla/servers-and-nodes/access/serial-console-and-rescue#fixing-the-os-means-reprovisioning).
3. Return the server to regular boot:
   - **Hetzner Cloud:** In the Cloud Console, ensure rescue mode is disabled, then power cycle the server.
   - **Bare Metal:** In Hetzner Robot, reset the server without activating rescue mode.
4. Once ready for automated management to resume, remove the pause annotation:
   ```console
   $ kubectl -n <cluster-namespace> annotate machine <machine-name> cluster.x-k8s.io/paused-
   ```

## Related

- [Collect logs with the log collector](/docs/hetzner/apalla/support/collect-logs-with-the-log-collector)
- [Grant Syself bare-metal access](/docs/hetzner/apalla/support/grant-syself-baremetal-access)
- [Serial console and rescue system](/docs/hetzner/apalla/servers-and-nodes/access/serial-console-and-rescue)
- [A server that will not provision](/docs/hetzner/apalla/servers-and-nodes/access/server-not-provisioning)
- [Where to start troubleshooting](/docs/hetzner/apalla/support/where-to-start)