The Syself Reverse Tunnel: From the Kubernetes API Server to the Nodes

The Syself Tunnel is a reverse tunnel that lets the Kubernetes API server reach back into a node. It's the path behind kubectl logs, kubectl exec, and kubectl port-forward, and behind the control plane scraping metrics from your pods. Each node dials out to the control plane and holds that connection open, so the API server never has to dial in and no node exposes an inbound port to the internet.

Note the direction: traffic runs from the control plane back into a node. It isn't the path you use to reach the cluster.

Why a node has to be reachable at all 

Kubernetes ships no built-in, secure way for the control plane to reach a node. Our nodes run directly on the public internet, and we don't want to open an inbound port on every node just so the API server can dial in.

The fix flips the direction. Instead of the control plane dialing the node, each node dials out to the control plane and holds that connection open. When the API server needs to reach a node, its traffic travels back down the connection the node already opened. Because nothing dials in, a node behind NAT or a strict firewall works with no special setup.

The tunnel runs over mTLS: both ends prove who they are with a certificate, and the traffic is encrypted end to end. Certificates are re-read on every handshake, so rotation needs no restart.

What the tunnel will and won't carry 

The tunnel has exactly two jobs. It reaches a node's own kubelet or an in-cluster address, and nothing else. It isn't a general-purpose proxy.

We kept that narrow on purpose. Any path the control plane can use to reach into a node is worth restricting hard, so the tunnel refuses everything outside its two jobs. Running kubectl exec never opens a new way into your nodes.

Why we built our own instead of konnectivity 

We ran the open-source konnectivity project for about five years, and helped maintain its releases for the last year and a half. Its generic design didn't fit this job well, and the project is barely maintained now. So we built a tunnel for the way we operate. It runs as two services inside the immutable node image instead of as static pods, which means a node never runs a mismatched agent and tunnel.

What changed in 1.36: a connection to every control plane 

The bigger change is where those connections go.

Before, a node held a single tunnel connection. A tunnel multiplexes many streams over one HTTP/2 connection, and that one connection sticks to a single control plane. That's true even with a load balancer in front, because a load balancer can't split one long-lived HTTP/2 connection across backends. So one control plane carried a node, and the others couldn't reach it directly.

Now each node opens a direct connection to every control plane. In an HA cluster with three or five control planes, any control plane can reach any node to fetch what it needs. This has two effects:

• A new or restarting control plane is reachable the moment it boots. That removes the flapping that used to show up when a control plane scaled up.
• Load spreads across the control planes instead of piling onto one, so rollouts and upgrades stay smooth. A dropped connection reconnects in seconds.

How it differs from the failover proxy 

The tunnel and the failover proxy are easy to confuse. The failover proxy keeps your workers talking to the control plane. The tunnel goes the other way, letting the control plane reach back into a node. You need both directions to operate and inspect a cluster at any time.

FAQ 

What breaks without it? 

kubectl logs, exec, and port-forward stop working, and the control plane can't scrape pod metrics. Your workloads keep serving, but you lose the ability to look inside and interact with them live.

Do I need to open ports on my nodes for this? 

No. Nodes dial outward and hold the connection open, so there's nothing inbound to expose.

Is the tunnel encrypted? 

Yes. It runs over mTLS, so both ends authenticate with certificates and the traffic is encrypted end to end.

The tunnel is how the control plane reaches your nodes. Controlling who reaches the control plane in the first place is the job of KubeGate.