Skip to main content

Serial console and rescue system

Inspect 1.36

When a node will not boot, you cannot reach it with kubectl or SSH. The serial console shows the node's boot output, including the kernel messages that never reach a log file. The Hetzner rescue system gives you a shell on the server to collect logs and, if needed, reprovision it.

Read the boot on the serial console

The serial console shows what happens before systemd (the Linux init system) starts: kernel panics and errors that never reach a log file.

A failing sealed boot looks like a verity error followed by an unreadable root:

text
		device-mapper: verity: 8:2: data block 0 is corrupted
EXT4-fs (dm-0): unable to read superblock
	

That is an integrity failure of the OS image. The node cannot repair it, so it must be reprovisioned. For other symptoms and their exact fix, see .

Activate the rescue system and collect logs

The Hetzner rescue system is a small Debian environment that boots entirely in RAM. It is the same environment Syself uses to install a node the first time.

To get a shell on a bare-metal server that will not boot normally:

Enable rescue

In the Hetzner Robot console, select the server, open the Rescue tab, choose your SSH key and keyboard layout, and activate the rescue system.

Reset into rescue

On the Reset tab, send a CTRL+ALT+DEL reset. The server reboots into the rescue system after a few minutes.

Mount the disk and collect logs

SSH into the rescue system with the key you selected, then mount the node's disk. Read /var/log/syself-activate.log with cat directly: it holds the full node activation output. journalctl -u syself-activate shows only the systemd lifecycle lines, not the actual output, so it is not enough on its own.

Caution

Collect what you need for the ticket before you reprovision. Reprovisioning overwrites the root disk.

Fixing the OS means reprovisioning

The fix for a broken sealed OS is to reprovision the node, which reinstalls a fresh, verified image. Data on the extra disks survives; the root disk is rewritten.

IPMI and KVM caveats

If the disk still fails the install check after reprovisioning, wipe it by hand from the rescue system: see .