Skip to main content

Emergency SSH access

SSH reaches a node directly, without going through the cluster. It works even before a node joins, when kubectl cannot reach it. Every node runs SSH on port 100, on by default, and every session leaves an audit trail.

When to use SSH

Reach for SSH when the problem is below Kubernetes: a node that will not join, a kubelet (the Kubernetes agent on each node) that is down, a boot that fails. When the node is healthy enough for the API server to answer, prefer collecting a bundle with the or kubectl debug node/<name>. Neither needs direct credentials, and both are easier to hand to support.

Note

If you , this method is not available on those nodes. See Debug a node when SSH is blocked.

Connect to a node

Find the node's IP, then SSH to it on port 100.

Find the node IP

		$ kubectl get node <node-name> -o jsonpath='{.status.addresses}'
	

Connect on port 100

Nodes run SSH on port 100 with key-only authentication. Password login is disabled.

		$ ssh -p 100 -i <your-key> -o IdentitiesOnly=yes root@<node-ip>
	

-o IdentitiesOnly=yes pairs with -i so the client offers only that one key. Without it, ssh tries every key your agent holds, and the node can cut the session off for too many failed attempts before it reaches the right one.

The keys allowed to log in are the cluster's configured SSH keys. For Hetzner Cloud nodes, that is the key named in the hcloud-ssh-key-name field of the hetzner Secret.

Important

Treat these keys as break-glass credentials: keys meant only for emergencies. Keep them somewhere secure, restrict them to the people who need them, and rotate them whenever you are unsure who holds a copy.

Debug a node when SSH is blocked

You can still reach a broken node with the SSH port closed.

Pause the machine

Pausing stops Syself Autopilot from replacing the node while you work on it.

		$ kubectl annotate machine <machine-name> cluster.x-k8s.io/paused=true
	

Boot into the rescue system and debug

Boot the node into the Hetzner rescue system and debug from there. See .

Remove the pause

		$ kubectl annotate machine <machine-name> cluster.x-k8s.io/paused-
	

What root can and cannot do

Root on a sealed node cannot change the OS. It is read-only and verified, so any edit is rejected. You can read files, inspect logs, and run diagnostics, but you cannot install packages or add a permanent service. A node that will not boot cannot be fixed in place either: rescue it with the , take what you need, and reprovision. This is the same boundary covered in full, with the exact commands, on .

Every session is audited

Root SSH sessions are recorded, including every command and any change to the SSH config, the emergency-access keys, and the cluster's kubeconfig. You can or to keep them off-node.