Serial console and rescue system
When a node will not boot, kubectl and even SSH are gone. You need output from before the operating system takes over. The serial console gives you that output. The Hetzner rescue system gives you a shell to collect logs and, if needed, reprovision.
Read the boot on the serial console#
The serial console shows what happens before systemd (the Linux init system) starts: kernel panics and dm-verity errors that never reach a log file. dm-verity (device-mapper verity, a Linux kernel feature) checks every disk block against a hash tree before use. A block that does not match will not mount.
A failing sealed boot looks like a verity error followed by an unreadable root:
device-mapper: verity: 8:2: data block 0 is corrupted
EXT4-fs (dm-0): unable to read superblock
That is an integrity failure of the OS image. The node cannot repair it, so it must be reprovisioned. For other symptoms and their exact fix, see Debug a node .
Activate the rescue system and collect logs#
The Hetzner rescue system is a small Debian environment that boots entirely in RAM. It is the same environment Syself uses to install a node the first time.
To get a shell on a bare-metal server that will not boot normally:
Enable rescue #
In the Hetzner Robot console, select the server, open the Rescue tab, choose your SSH key and keyboard layout, and activate the rescue system.
Reset into rescue #
On the Reset tab, send a CTRL+ALT+DEL reset. The server reboots into the rescue system after a few minutes.
Mount the disk and collect logs #
SSH into the rescue system with the key you selected, then mount the node's disk. Read /var/log/syself-activate.log with cat directly: it holds the full node activation output. journalctl -u syself-activate shows only the systemd lifecycle lines, not the actual output, so it is not enough on its own.
Caution
Collect what you need for the ticket before you reprovision. Reprovisioning overwrites the root disk.
Fixing the OS means reprovisioning#
The fix for a broken sealed OS is to reprovision the node, which reinstalls a fresh, verified image. Data on the extra disks survives; the root disk is rewritten.
Reprovisioning runs the same install the node had the first time. The provisioner writes the sealed image to disk, then checks the written bytes against the build-time hash before handing the node back to Kubernetes. A worker that reprovisions cleanly rejoins the cluster on its own once the install finishes.
Important
There is no verity-off boot entry. You cannot "boot without the seal" to edit the OS back to health, so a broken sealed OS is always fixed by reprovisioning.
IPMI and KVM caveats#
Out-of-band management is not universal on Hetzner dedicated servers. Since 2019, most models no longer ship IPMI/KVM modules (IPMI: a standard for remote server management). Only Dell models and Server Auction servers have out-of-band management. Where it is not available, the serial console plus the rescue system are your tools.
Even on a server without a built-in module, you can order a KVM console from Hetzner. They attach a remote console on request, free for up to three hours, which is enough to watch a boot or change a BIOS setting. Request it from Robot; see Hetzner's KVM console docs.
Where a server does have a BMC (Baseboard Management Controller, the chip behind IPMI/KVM), reach it through the IPMI web interface, or with the vendor tool that Hetzner documents. See Hetzner's IPMI docs for the steps.
A server that will not even reach rescue is a case to get provisioning unstuck or grant Syself bare-metal access .
If the disk still fails the install check after reprovisioning, wipe it by hand from the rescue system: see Wipe a disk before provisioning .