Skip to main content

Platform Components: KubeGate, syself-agent, and the Tunnel

Inspect 1.36

Syself Autopilot puts every node straight on the public internet, and a small set of Syself-built pieces makes that safe. Why a node is safe there is the ; why there is no private network to hide behind is the . This page covers the pieces themselves.

The thing you most need to protect is the API server, the entry point every kubectl command talks to. KubeGate and syself-agent protect it:

  • KubeGate, a gateway that listens on the node's public address and decides who may open a connection to the API server.
  • syself-agent, one program on every server. Each service it runs is a command inside that one binary: the node agent's health checks, the worker failover proxy, and a reverse tunnel. With the tunnel, every node dials outward, so the control plane never needs to dial in.

Both ship inside the and share its version, so a node never runs a mismatched agent and tunnel.

KubeGate is a separate program from syself-agent, and it guards the load-balancer path in front of the API server, so it comes first.

KubeGate, the gateway in front of the API server

On a control-plane node, the API server listens only on the node's loopback address, the address a machine uses to talk to itself. Nothing off the node can reach it there. KubeGate runs on the same node as a static pod, a pod the node starts on its own. It listens on the node's public address and forwards each connection to the API server.

KubeGate never decrypts. It passes encrypted traffic straight through, and the API server presents its own certificate end to end and authenticates every connection.

KubeGate runs as an unprivileged user. It holds no credential or key, and it cannot issue or sign anything. The cluster CA signing key is readable only by root, out of its reach.

There are two ways to reach the API server, each with its own filter:

Path Who arrives Filtered by
Direct, node to control plane only cluster nodes , which recognizes the other node as a cluster member
Through the load balancer anyone on the internet KubeGate, on the real client address (the load balancer adds a small prefix to each connection, the PROXY protocol header, that carries the client's real address)

A registered node has a Cilium identity and takes the direct path. KubeGate forwards it without gating, because Cilium already decided it on a signal KubeGate cannot see. The load-balancer path is open to the world. That is the path KubeGate checks.

The admission check

On the gated load-balancer path, KubeGate decides who may open a connection to the API server.

flowchart LR
  client["Internet client"] -->|"encrypted"| lb["Control-plane load balancer<br/>adds the real client address"]
  join["Joining node<br/>(not a cluster member yet)"] --> lb
  mgmt["Management cluster"] --> lb
  lb --> kg["KubeGate<br/>public address, unprivileged"]
  reg["Registered node"] -->|"direct, recognized as a cluster member"| kg
  kg -->|"passed through, still encrypted"| api["API server<br/>loopback only"]

On the gated path, KubeGate lets in cluster nodes and the platform's own traffic. Your address allowlist applies only to plain internet traffic, so the management cluster and joining nodes get in whatever your allowlist says. Getting through the gate only opens a connection; the API server still authenticates it like any other client.

The limits of the gate

KubeGate gates connections that arrive from the internet. A caller already on the node skips it: a pod on the node's own network, or root over SSH, can reach the API server on its loopback address directly, without passing through the gate.

Skipping the gate grants no access. The API server still authenticates every request, so a caller on the node needs a valid identity like any other, and the gate never was the thing keeping them out. The only effect is on the audit log. A request that arrives on the node's loopback address is recorded as coming from the node itself, so it cannot be told apart from a control-plane component's own traffic. Cryptography cannot fix that, because the request genuinely originates on the node. The backstop is an alert on audit events that look like they come from the node's own loopback address.

Putting the API server behind KubeGate also removes the client's identity from the audit log. Every connection now arrives from the loopback address, so the log can no longer tell who the client was. A helper in the node agent restores the real client address into the audit log. It sets this locally on the node, after the connection has passed KubeGate, so a remote client cannot forge it.

The helper fails open. Reachability never waits on it, so if it is degraded the connection is served anyway and a marker is recorded instead. One marker means KubeGate handled the connection but could not name the caller.

IPv6 clients are the one case the helper cannot name at all. Publishing only an IPv4 address for your control-plane endpoint is the recommendation. The nodes use IPv4 on the loopback address, so an IPv6 client is served and recorded with a different marker. With IPv6 you keep reachability but lose the client's identity in the log, and Hetzner load balancers carry both IPv4 and IPv6 by default.

Keeping the gate working

Two operational rules keep the gate doing its job. First, the address allowlist:

Warning

Two ways to break the address allowlist. Enable it only after PROXY protocol is live on the load balancer (the cluster reports it as enabled); before that KubeGate cannot see real client addresses and the check can be bypassed. And keep the trusted-address list to the load balancer's own address alone: an address listed both as trusted and inside an allowlist's range lets any client in that range choose the address the allowlist sees.

Second, remember what KubeGate is. It decides who may open a connection. It does not authenticate anyone; RBAC, the Kubernetes permission system, still decides what an admitted connection may do. once PROXY protocol is confirmed live, and alert on audit events attributed to the loopback address, because a legitimate client always leaves a real address or a marker. A plain loopback address with no marker means the connection never crossed the gate.

The services inside syself-agent

syself-agent is built into the read-only image and covered by dm-verity like the rest of the disk. Each service runs as its own background process, so a failing one restarts by itself. These are the services it runs:

Service Runs on Job
Node agent every node keeps the local list of the cluster's nodes, runs the health checks that set node conditions, repairs what it can in place, and turns on the failover proxy
Tunnel agent every node dials the reverse tunnel out to each control plane and holds it open
Tunnel server control planes accepts tunnels and gives the local API server a path to every node
Failover proxy (syself-proxy) workers the kubelet failover proxy, reachable only from the worker itself

One job is not on this list: joining a node to the cluster. That runs outside syself-agent, as a separate first-boot program that exits once the node has joined.

The conditions the node agent sets are the signal the platform acts on to .

The reverse tunnel

kubectl logs, exec, and port-forward all need the API server to reach a node's kubelet. The kubelet is the agent Kubernetes runs on each node to start and watch containers.

Instead of the API server dialing the node, the node dials out. Each node runs a tunnel agent that dials every control plane over mutual TLS (mTLS, where both sides prove their identity with certificates) and holds the connection open. A tunnel server on the control plane accepts the connection and gives the API server a local path into it. When the API server needs a node, its traffic goes back down the tunnel the node already opened.

The API server never dials a node, so nodes behind NAT or a strict firewall work unchanged.

sequenceDiagram
  participant API as API server
  participant S as Tunnel server (control plane)
  participant A as Tunnel agent (node)
  participant K as kubelet (node)

  Note over A,S: at boot, and held open from then on
  A->>S: dials out (mTLS, cluster CA)
  Note over API: later: kubectl logs for a pod on this node
  API->>S: node-bound request
  S->>A: sent down the tunnel the node opened
  A->>K: local connection to the kubelet only
  K-->>API: response returns the same way

The tunnel only reaches the local kubelet or an in-cluster address. It is not a general-purpose proxy.

The failover proxy

A worker normally reaches the API server through one control-plane load balancer. Lose that load balancer for about forty seconds and the kubelet goes NotReady, Kubernetes moves the pods, and one load-balancer failure takes every worker down at once. The failover proxy, syself-proxy, removes that single point of failure.

It runs on the worker and is reachable only from that worker. It forwards kubelet traffic to the control planes and falls back to direct control-plane connections when the load balancer fails.

A check inside the node agent runs about every fifteen seconds. The check proves the full path through the proxy with the kubelet's own credentials, then points the kubelet at the local proxy and restarts it. The connection still verifies, because the API server's certificate also covers the loopback address.

Control planes skip the proxy, since their kubelet already talks to the local API server.

What listens on a node

Every listener on a node falls into one of two groups: unreachable off the node, or reachable but authenticated. Nothing is reachable without authentication. KubeGate is the one public listener; it passes encrypted traffic through and the API server authenticates it. The tunnel server on control planes takes only cluster members, over mTLS behind the host firewall. Everything else, the API server, the failover proxy, and the tunnel's health and metrics, listens on the node itself and is unreachable from off the node. One more listener answers only the load balancer: a readiness check that reports whether the local API server is serving, so the load balancer sends traffic to a control plane only once it is ready.

If an attacker takes over one piece

Take over any single reachable piece and you get only what that piece does:

Piece taken What it yields
KubeGate a raw-TLS forwarder; no credential, no CA key, no capabilities
the tunnel agent a route to the local kubelet on that node and to in-cluster addresses; every other connection is refused

How this adds up to a node being safe on the public internet is the , which needs .

How this connects to the two-cluster model

The sits outside your cluster, so it reaches your API server over the public control-plane endpoint, the path KubeGate guards. The reverse tunnel runs inside your cluster: each node dials out to its own control planes so the API server can reach a kubelet without dialing in. The guards the direct node-to-node path KubeGate never sees, and nodes are on public addresses at all because the platform needs .

Next: .