The Syself Reverse Tunnel: From the Kubernetes API Server to the Nodes

Note

This resource is part of the Syself Autopilot 1.36 release. For the whole picture, see what's new in 1.36.

The Syself Tunnel is a reverse tunnel that lets the Kubernetes API server reach back into a node. It's the path behind kubectl logs, kubectl exec, and kubectl port-forward, and behind the control plane scraping metrics from your pods. Each node dials out to the control plane and holds that connection open, so the API server never has to dial in and no node exposes an inbound port to the internet.

The direction catches people out. Traffic runs from the control plane back into a node, not from you into the cluster.

Why a node has to be reachable at all

Kubernetes ships no built-in, secure way for the control plane to reach a node. And our nodes run straight on the public internet, so the last thing we want is to punch an inbound hole in every one so the API server can dial in.

The fix flips the direction. Instead of the control plane dialing the node, each node dials out to the control plane and holds that connection open. When the API server needs to reach a node, its traffic travels back down the connection the node already opened. Because nothing dials in, a node behind NAT or a strict firewall works with no special setup.

The tunnel runs over mTLS — both ends prove who they are with a certificate, and the traffic is encrypted end to end. Certificates are re-read on every handshake, so rotation needs no restart.

What the tunnel will and won't carry

The tunnel has exactly two jobs. It reaches a node's own kubelet or an in-cluster address, and nothing else. It isn't a general-purpose proxy.

Where the request is headed	What happens
The node's own address	Routed to that node's local kubelet
An in-cluster pod or service	Routed inside the cluster
Anywhere else	Refused

We kept that narrow on purpose. Any path the control plane can use to reach into a node is worth restricting hard, so the tunnel refuses everything outside its two jobs. Running kubectl exec never opens a new way into your nodes.

Why we built our own instead of konnectivity

We ran the open-source konnectivity project for about five years, and helped maintain its releases for the last year and a half. Its generic design never quite fit this job, and it's barely maintained now. So we built a tunnel for how we actually operate. It runs as two services inside the immutable node image instead of as static pods, which means a node never runs a mismatched agent and tunnel.

What changed in 1.36: a connection to every control plane

The bigger shift is where those connections go.

Before, a node held a single tunnel connection. A tunnel multiplexes many streams over one HTTP/2 connection, and that one connection sticks to a single control plane — even with a load balancer in front, because a load balancer can't split one long-lived HTTP/2 connection across backends. So one control plane carried a node, and the others couldn't reach it directly.

Now each node opens a direct connection to every control plane. In an HA cluster with three or five control planes, any control plane can reach any node to fetch what it needs. That buys two things:

A new or restarting control plane is reachable the moment it boots. That removes the flapping that used to show up when a control plane scaled up.
Load spreads across the control planes instead of piling onto one, so rollouts and upgrades stay smooth. A dropped connection reconnects in seconds.

How it differs from the failover proxy

People mix these two up. The failover proxy keeps your workers talking to the control plane. The tunnel goes the other way, letting the control plane reach back into a node. Two directions, and you need both to keep a cluster you can always operate and inspect.

FAQ

What breaks without it?

kubectl logs, exec, and port-forward stop working, and the control plane can't scrape pod metrics. Your workloads keep serving, but you lose the ability to look inside and interact with them live.