Emergency SSH access

Inspect 1.36

SSH reaches a node directly, without going through the cluster. It works even before a node joins, when kubectl cannot reach it. Every node runs SSH on port 100, on by default, and every session leaves an audit trail.

When to use SSH#

Reach for SSH when the problem is below Kubernetes: a node that will not join, a kubelet (the Kubernetes agent on each node) that is down, a boot that fails. When the node is healthy enough for the API server to answer, prefer collecting a bundle with the log collector or kubectl debug node/<name>. Neither needs direct credentials, and both are easier to hand to support.

Note

If you restricted or blocked SSH , this method is not available on those nodes. See Debug a node when SSH is blocked.

Connect to a node#

Find the node's IP, then SSH to it on port 100.

Find the node IP #

		$ kubectl get node <node-name> -o jsonpath='{.status.addresses}'

Connect on port 100 #

Nodes run SSH on port 100 with key-only authentication. Password login is disabled.

		$ ssh -p 100 -i <your-key> -o IdentitiesOnly=yes root@<node-ip>

-o IdentitiesOnly=yes pairs with -i so the client offers only that one key. Without it, ssh tries every key your agent holds, and the node can cut the session off for too many failed attempts before it reaches the right one.

The keys allowed to log in are the cluster's configured SSH keys. For Hetzner Cloud nodes, that is the key named in the hcloud-ssh-key-name field of the hetzner Secret.

Important

Treat these keys as break-glass credentials: keys meant only for emergencies. Keep them somewhere secure, restrict them to the people who need them, and rotate them whenever you are unsure who holds a copy.

Lock SSH down#

SSH is not used for provisioning or at runtime, so you can restrict it as much as you want.

Only two ports on a node face the public internet: SSH on port 100, and the Kubernetes API on 6443. You reach the API through KubeGate, which is policy-driven, so you decide who gets to it by writing access policies (see Manage access and tenancy ). SSH is the one you lock down here. For the full port list, see Ports and listeners .

Option	Effect
Block the port	Close port 100 completely. Nothing at runtime breaks.
Allow only a bastion or specific IPs	A bastion is a single hardened host that is your one entry point for SSH. Limit port 100 to its address, or to a set of trusted IP ranges.

You restrict SSH by adding your own Cilium policy with an ingressDeny on port 100. A deny always wins over the platform's default allow, so you never edit the built-in rule (it is reconciled, and manual edits are reverted). This example allows SSH only from your management network and denies the rest:

yaml

		apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: ssh-lockdown
spec:
  nodeSelector:
    matchLabels: {} # all nodes
  ingressDeny:
    - fromCIDRSet:
        - cidr: 0.0.0.0/0
          except:
            - 10.0.0.0/8 # your trusted IP range(s)
      toPorts:
        - ports:
            - port: "100"
              protocol: TCP

To close SSH completely, deny 0.0.0.0/0 with no except. See Restrict SSH access for the full how-to.

Tip

Confirm a block took effect: from a node shell, run ss -tlnp and check port 100 is gone. Or try to connect from outside the cluster and watch the connection time out.

Note

SSH is on by default and gives you a live shell on a running node. You can restrict it to a bastion host, turn it off, or re-enable it only when you need to inspect a node. See Restrict SSH access . With SSH closed, the fallback is booting the server into the Hetzner rescue system, but the node is not running while you are in rescue, so that path is for repair and debugging rather than looking at a live node.

Debug a node when SSH is blocked#

You can still reach a broken node with the SSH port closed.

Pause the machine #

Pausing stops Syself Autopilot from replacing the node while you work on it.

		$ kubectl annotate machine <machine-name> cluster.x-k8s.io/paused=true

Boot into the rescue system and debug #

Boot the node into the Hetzner rescue system and debug from there. See Serial console and rescue system .

Remove the pause #

		$ kubectl annotate machine <machine-name> cluster.x-k8s.io/paused-

What root can and cannot do#

Root on a sealed node cannot change the OS. It is read-only and verified, so any edit is rejected. You can read files, inspect logs, and run diagnostics, but you cannot install packages or add a permanent service. A node that will not boot cannot be fixed in place either: rescue it with the serial console and Hetzner rescue system , take what you need, and reprovision. This is the same boundary covered in full, with the exact commands, on What you can and cannot change on a sealed node .

Every session is audited#

Every root SSH session is recorded. Each command is logged, and changes to sensitive files (the SSH configuration, the emergency-access keys, and the cluster's Kubernetes config) are captured too. Emergency access always leaves an audit trail, by design. See retrieve audit logs to read those records.

Machine health checks and remediation

Syself Autopilot watches your nodes and replaces the ones that go unhealthy on their own, unless you pause it first for planned maintenance.

Debug a node

Get onto a node, read its logs, and work out what's a real problem versus routine noise.