Debug node networking

Most node-level network problems appear in a few files and service logs on the node. Check those first, then match what you see to the failures below and resolve it. To get onto a node in the first place, see Emergency SSH access .

Where to look first#

The node's view of the cluster is one file:

		$ cat /var/lib/syself/nodeinfo/nodes.json
	

It holds the node's current list of control-plane and worker IPs, the cluster CIDRs (the pod and Service ranges), and whether the failover proxy is active. (The failover proxy is the kubelet's backup route to the API server when the load balancer path fails, covered below.) Most problems appear here first. The control-plane load balancer address is in a separate file, /etc/kubernetes/control-plane-endpoint.

The service logs cover the rest:

		$ journalctl -u syself-agent -b          # writes nodes.json, runs health checks, activates the proxy
$ journalctl -u syself-tunnel-agent -b   # the tunnel from the node to the control plane
$ journalctl -u syself-tunnel-server -b  # control-plane nodes only: the server side of the tunnel
$ journalctl -u syself-proxy -b          # worker nodes only: the kubelet failover proxy
	

SSH is on port 100, not 22:

		$ ssh -p 100 root@<node-ip>
	

The API server cannot reach a pod#

Symptom: kubectl top nodes times out, or an admission webhook times out. The tunnel agent logs show is outside the cluster CIDRs [].

Cause: the tunnel refuses to route to a pod because it has no cluster CIDRs. The clusterCIDRs field in nodes.json is empty.

		$ cat /var/lib/syself/nodeinfo/nodes.json   # check clusterCIDRs
$ cat /etc/kubernetes/cluster-network        # should hold POD_CIDRS= and SERVICE_CIDRS=
	

Fix: if /etc/kubernetes/cluster-network is missing or empty, the node was not provisioned from a current release; reprovision it. If the file is present but clusterCIDRs is still empty, syself-agent has not read it yet: restart the service and watch its log.

Workers go NotReady when the load balancer fails#

Symptom: every worker flips to NotReady at the same time whenever the control-plane load balancer has trouble.

Cause: the kubelet is still communicating with the load balancer directly. The failover proxy, which keeps the kubelet connected when the load balancer path fails, never activated. When it is active, the kubelet reaches the API server through https://127.0.0.1:7443 on the node instead of the load balancer.

		$ grep server: /etc/kubernetes/kubelet.conf   # want https://127.0.0.1:7443, not the LB address
$ ss -tlnp | grep 7443                         # is the proxy listening?
	

Fix: check journalctl -u syself-agent -b | grep -i proxy. The proxy activates a few minutes after a node joins, once nodes.json has control-plane IPs. If it is not listening on 7443, or nodes.json has no control-plane IPs yet, wait and recheck; if it never activates, reprovision the node.

The tunnel agent will not connect#

Symptom: journalctl -u syself-tunnel-agent -b shows repeated connection failures to port 8180, or the agent has no open tunnels (open_server_connections is 0 at 127.0.0.1:8182/metrics).

Cause: the agent knows no control-plane addresses, or its mTLS is failing.

		$ cat /var/lib/syself/nodeinfo/nodes.json | grep controlPlanes
$ ls /var/lib/kubelet/pki/kubelet-client-current.pem   # the client cert for mTLS
$ ls /etc/kubernetes/pki/ca.crt                         # the CA it verifies the server against
	

Fix: if controlPlanes is empty, syself-agent has not synced yet; wait a few minutes and check its log. If the kubelet certificate is missing or expired, that is a certificate problem, not a tunnel problem: the agent re-reads the certificate on every handshake, so it recovers automatically once the kubelet rotates it.

scp fails to copy a file#

Symptom: ssh -p 100 provides a shell, but scp -P 100 file root@<node-ip>:/root/ quits immediately with Connection closed.

Cause: the node's SSH server does not run the SFTP subsystem. A modern scp uses the SFTP protocol by default, so it has nothing to connect to.

Fix: force the legacy SCP protocol with -O, or pipe the file over ssh:

		$ scp -O -P 100 file root@<node-ip>:/root/
$ ssh -p 100 root@<node-ip> 'cat > /root/file' < file
	

Port 6443 is refused from the internet#

Connecting directly to a control-plane node on port 6443 from a public host is refused, even while kubectl works normally.

		$ kubectl get ciliumclusterwidenetworkpolicy control-plane-apiserver-lockdown -o yaml
	

Debug from inside the cluster with a throwaway pod#

Node-level checks miss problems that only exist on the pod network. To see exactly what your application sees, run a temporary pod with netshoot⁠, an image that bundles the usual tools: dig, curl, traceroute, tcpdump, mtr, and iperf:

		$ kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash
	

From that shell, dig a Service name to test DNS, curl another pod or Service to test reachability, and traceroute an external host to test egress, each from the same vantage point your workloads have; --rm removes the pod when you exit. To inspect a specific pod's network namespace instead, attach an ephemeral container to it: kubectl debug -it <pod> --image=nicolaka/netshoot.

Where to go next#

• Diagnose pod DNS resolution
• Run a Cilium connectivity test
• Ports and listeners