SSH reaches a node directly, without going through the cluster. It works even before a node joins, when `kubectl` cannot reach it. Every node runs SSH on port 100, on by default, and every session leaves an audit trail. ## When to use SSH Reach for SSH when the problem is below Kubernetes: a node that will not join, a kubelet (the Kubernetes agent on each node) that is down, a boot that fails. When the node is healthy enough for the API server to answer, prefer collecting a bundle with the [log collector](/docs/hetzner/apalla/support/collect-logs-with-the-log-collector) or `kubectl debug node/`. Neither needs direct credentials, and both are easier to hand to support. > [!NOTE] > If you [restricted or blocked SSH](/docs/hetzner/apalla/security/restrict-ssh-access), this method is not available on those nodes. See [Debug a node when SSH is blocked](#debug-a-node-when-ssh-is-blocked). ## Connect to a node Find the node's IP, then SSH to it on port 100. Find the node IP ```console $ kubectl get node -o jsonpath='{.status.addresses}' ``` Connect on port 100 Nodes run SSH on **port 100** with key-only authentication. Password login is disabled. ```console $ ssh -p 100 -i -o IdentitiesOnly=yes root@ ``` `-o IdentitiesOnly=yes` pairs with `-i` so the client offers only that one key. Without it, ssh tries every key your agent holds, and the node can cut the session off for too many failed attempts before it reaches the right one. The keys allowed to log in are the cluster's configured SSH keys. For Hetzner Cloud nodes, that is the key named in the `hcloud-ssh-key-name` field of the `hetzner` Secret. > [!IMPORTANT] > Treat these keys as break-glass credentials: keys meant only for emergencies. Keep them somewhere secure, restrict them to the people who need them, and rotate them whenever you are unsure who holds a copy. ## Debug a node when SSH is blocked You can still reach a broken node with the SSH port closed. Pause the machine Pausing stops Syself Autopilot from replacing the node while you work on it. ```console $ kubectl annotate machine cluster.x-k8s.io/paused=true ``` Boot into the rescue system and debug Boot the node into the Hetzner rescue system and debug from there. See [Serial console and rescue system](/docs/hetzner/apalla/servers-and-nodes/access/serial-console-and-rescue). Remove the pause ```console $ kubectl annotate machine cluster.x-k8s.io/paused- ``` ## What root can and cannot do Root on a sealed node cannot change the OS. It is read-only and verified, so any edit is rejected. You can read files, inspect logs, and run diagnostics, but you cannot install packages or add a permanent service. A node that will not boot cannot be fixed in place either: rescue it with the [serial console and Hetzner rescue system](/docs/hetzner/apalla/servers-and-nodes/access/serial-console-and-rescue), take what you need, and reprovision. This is the same boundary covered in full, with the exact commands, on [What you can and cannot change on a sealed node](/docs/hetzner/apalla/servers-and-nodes/node-os/what-you-can-change-on-a-sealed-node). ## Every session is audited Root SSH sessions are recorded, including every command and any change to the SSH config, the emergency-access keys, and the cluster's kubeconfig. You can [retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) or [ship audit logs to a SIEM](/docs/hetzner/apalla/security/ship-audit-logs-to-a-siem) to keep them off-node. ## Related - [Restrict SSH access](/docs/hetzner/apalla/security/restrict-ssh-access) - [Serial console and rescue system](/docs/hetzner/apalla/servers-and-nodes/access/serial-console-and-rescue) - [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) - [Debug a node](/docs/hetzner/apalla/servers-and-nodes/access/debug-a-node)