Most node-level network problems appear in a few files and service logs on the node. Check those first, then match what you see to the failures below and resolve it. To get onto a node in the first place, see [Emergency SSH access](/docs/hetzner/apalla/servers-and-nodes/access/emergency-ssh-access). | Symptom | Likely cause | Go to | | ------------------------------------------------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `kubectl top` or an admission webhook times out | the tunnel has no cluster CIDRs | [The API server cannot reach a pod](#the-api-server-cannot-reach-a-pod) | | Every worker turns `NotReady` at once | the failover proxy never activated | [Workers go NotReady when the load balancer fails](#workers-go-notready-when-the-load-balancer-fails) | | The tunnel agent retries and never connects | no control-plane IPs, or mTLS is failing | [The tunnel agent will not connect](#the-tunnel-agent-will-not-connect) | | `scp` closes with `Connection closed` | the SFTP subsystem is off | [scp fails to copy a file](#scp-fails-to-copy-a-file) | | Port `6443` is refused from the internet | the apiserver lockdown, functioning as intended | [Port 6443 is refused from the internet](#port-6443-is-refused-from-the-internet) | ## Where to look first **The node's view of the cluster** is one file: ```console $ cat /var/lib/syself/nodeinfo/nodes.json ``` It holds the node's current list of control-plane and worker IPs, the cluster CIDRs (the pod and Service ranges), and whether the failover proxy is active. (The failover proxy is the kubelet's backup route to the API server when the load balancer path fails, covered below.) Most problems appear here first. The control-plane load balancer address is in a separate file, `/etc/kubernetes/control-plane-endpoint`. **The service logs** cover the rest: ```console $ journalctl -u syself-agent -b # writes nodes.json, runs health checks, activates the proxy $ journalctl -u syself-tunnel-agent -b # the tunnel from the node to the control plane $ journalctl -u syself-tunnel-server -b # control-plane nodes only: the server side of the tunnel $ journalctl -u syself-proxy -b # worker nodes only: the kubelet failover proxy ``` SSH is on port `100`, not `22`: ```console $ ssh -p 100 root@ ``` ## The API server cannot reach a pod **Symptom:** `kubectl top nodes` times out, or an admission webhook times out. The tunnel agent logs show `is outside the cluster CIDRs []`. **Cause:** the tunnel refuses to route to a pod because it has no cluster CIDRs. The `clusterCIDRs` field in `nodes.json` is empty. ```console $ cat /var/lib/syself/nodeinfo/nodes.json # check clusterCIDRs $ cat /etc/kubernetes/cluster-network # should hold POD_CIDRS= and SERVICE_CIDRS= ``` **Fix:** if `/etc/kubernetes/cluster-network` is missing or empty, the node was not provisioned from a current release; reprovision it. If the file is present but `clusterCIDRs` is still empty, `syself-agent` has not read it yet: restart the service and watch its log. ## Workers go NotReady when the load balancer fails **Symptom:** every worker flips to `NotReady` at the same time whenever the control-plane load balancer has trouble. **Cause:** the kubelet is still communicating with the load balancer directly. The failover proxy, which keeps the kubelet connected when the load balancer path fails, never activated. When it is active, the kubelet reaches the API server through `https://127.0.0.1:7443` on the node instead of the load balancer. ```console $ grep server: /etc/kubernetes/kubelet.conf # want https://127.0.0.1:7443, not the LB address $ ss -tlnp | grep 7443 # is the proxy listening? ``` **Fix:** check `journalctl -u syself-agent -b | grep -i proxy`. The proxy activates a few minutes after a node joins, once `nodes.json` has control-plane IPs. If it is not listening on `7443`, or `nodes.json` has no control-plane IPs yet, wait and recheck; if it never activates, reprovision the node. ## The tunnel agent will not connect **Symptom:** `journalctl -u syself-tunnel-agent -b` shows repeated connection failures to port `8180`, or the agent has no open tunnels (`open_server_connections` is 0 at `127.0.0.1:8182/metrics`). **Cause:** the agent knows no control-plane addresses, or its mTLS is failing. ```console $ cat /var/lib/syself/nodeinfo/nodes.json | grep controlPlanes $ ls /var/lib/kubelet/pki/kubelet-client-current.pem # the client cert for mTLS $ ls /etc/kubernetes/pki/ca.crt # the CA it verifies the server against ``` **Fix:** if `controlPlanes` is empty, `syself-agent` has not synced yet; wait a few minutes and check its log. If the kubelet certificate is missing or expired, that is a certificate problem, not a tunnel problem: the agent re-reads the certificate on every handshake, so it recovers automatically once the kubelet rotates it. ## scp fails to copy a file **Symptom:** `ssh -p 100` provides a shell, but `scp -P 100 file root@:/root/` quits immediately with `Connection closed`. **Cause:** the node's SSH server does not run the SFTP subsystem. A modern `scp` uses the SFTP protocol by default, so it has nothing to connect to. **Fix:** force the legacy SCP protocol with `-O`, or pipe the file over `ssh`: ```console $ scp -O -P 100 file root@:/root/ $ ssh -p 100 root@ 'cat > /root/file' < file ``` ## Port 6443 is refused from the internet Connecting directly to a control-plane node on port `6443` from a public host is refused, even while `kubectl` works normally. > [!NOTE] > This is expected, not a fault. The apiserver lockdown policy denies `6443` from the internet, with the control-plane load balancer as the only exception. Cluster-internal sources are matched by Cilium identity (by pod labels, not by IP), so they are unaffected. Reach the API through the load balancer as normal. ```console $ kubectl get ciliumclusterwidenetworkpolicy control-plane-apiserver-lockdown -o yaml ``` If the policy is missing, contact Syself support. ## Debug from inside the cluster with a throwaway pod Node-level checks miss problems that only exist on the pod network. To see exactly what your application sees, run a temporary pod with [netshoot](https://github.com/nicolaka/netshoot), an image that bundles the usual tools: `dig`, `curl`, `traceroute`, `tcpdump`, `mtr`, and `iperf`: ```console $ kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash ``` From that shell, `dig` a Service name to test DNS, `curl` another pod or Service to test reachability, and `traceroute` an external host to test egress, each from the same vantage point your workloads have; `--rm` removes the pod when you exit. To inspect a specific pod's network namespace instead, attach an ephemeral container to it: `kubectl debug -it --image=nicolaka/netshoot`. ## Where to go next - [Diagnose pod DNS resolution](/docs/hetzner/apalla/network/debug/diagnose-pod-dns-resolution) - [Run a Cilium connectivity test](/docs/hetzner/apalla/network/debug/run-a-cilium-connectivity-test) - [Ports and listeners](/docs/hetzner/apalla/reference/ports-and-listeners)