Skip to main content

Platform Components: KubeGate, syself-agent, and the Tunnel

Inspect 1.36

Syself Autopilot puts every node straight on the public internet, and a small set of Syself-built pieces makes that safe. Why a node is safe there is the ; why there is no private network to hide behind is the . This page covers the pieces themselves.

The thing you most need to protect is the API server, the entry point every kubectl command talks to. KubeGate and syself-agent protect it:

  • KubeGate, a gateway that listens on the node's public address and decides who may open a connection to the API server.
  • syself-agent, one program on every server. Each service it runs is a command inside that one binary: the node agent's health checks, the worker failover proxy, and a reverse tunnel. With the tunnel, every node dials outward, so the control plane never needs to dial in.

Both ship inside the and share its version, so a node never runs a mismatched agent and tunnel.

KubeGate is a separate program from syself-agent, and it guards the load-balancer path in front of the API server, so it comes first.

KubeGate, the gateway in front of the API server#

On a control-plane node, the API server listens only on the node's loopback address, the address a machine uses to talk to itself. Nothing off the node can reach it there. KubeGate runs on the same node as a static pod, a pod the node starts on its own. It listens on the node's public address and forwards each connection to the API server.

KubeGate never decrypts. It passes encrypted traffic straight through, and the API server presents its own certificate end to end and authenticates every connection.

KubeGate runs as an unprivileged user. It holds no credential or key, and it cannot issue or sign anything. The cluster CA signing key is readable only by root, out of its reach.

There are two ways to reach the API server, each with its own filter:

Path Who arrives Filtered by
Direct, node to control plane only cluster nodes , which recognizes the other node as a cluster member
Through the load balancer anyone on the internet KubeGate, on the real client address (the load balancer adds a small prefix to each connection, the PROXY protocol header, that carries the client's real address)

A registered node has a Cilium identity and takes the direct path. KubeGate forwards it without gating, because Cilium already decided it on a signal KubeGate cannot see. The load-balancer path is open to the world. That is the path KubeGate checks.

The admission check#

On the gated load-balancer path, KubeGate decides who may open a connection to the API server.

flowchart LR
  client["Internet client"] -->|"encrypted"| lb["Control-plane load balancer<br/>adds the real client address"]
  join["Joining node<br/>(not a cluster member yet)"] -->|"node ticket"| lb
  mgmt["Management cluster"] -->|"egress ticket"| lb
  lb --> kg["KubeGate<br/>public address, unprivileged"]
  reg["Registered node"] -->|"direct, recognized as a cluster member"| kg
  kg -->|"passed through, still encrypted"| api["API server<br/>loopback only"]

On the gated path, KubeGate lets in cluster nodes and anything carrying a valid ticket. Your address allowlist applies only to plain internet traffic that carries no ticket. A ticket takes priority over the allowlist, so the management cluster and joining nodes get in whatever your allowlist says. A ticket only lets a connection through the gate; the API server still authenticates it like any other client.

Short-lived gate tickets#

The joining node and the management cluster both need to cross the gated load-balancer path without being recognized as a cluster member. Each gets its own short-lived ticket, signed by the cluster CA.

Node ticket Egress ticket
For a joining node, before it is a cluster member the management cluster reaching your workload API server
Produced by the bootstrap controller, into the machine's bootstrap data the management cluster, in memory
Stored on the node's disk, readable only by the root user never on disk; added to each connection
Lifetime two hours two hours; held in memory and re-issued before it expires

The node ticket covers the join. Every machine is marked to receive one, which tells the bootstrap controller to sign a ticket and write it into the node's bootstrap data. The joining node presents it at the start of each connection, and KubeGate checks it fresh every time and stores nothing. It is a bearer token, so its short lifetime matters. A node has two hours from the moment its bootstrap data is prepared to join. KubeGate rejects any ticket signed to last longer.

A ticket only buys a connection, not trust. Getting through KubeGate is permission to make a request, not authentication. The joining node still has to prove who it is to the API server, and it earns its real credentials through a certificate signing request that a Syself controller reviews. The controller signs a request only for a node that is genuinely joining. So a stolen ticket gets an attacker a connection and nothing else: with no approved certificate it has no identity, so it cannot step in as a node even while it holds the ticket. This is what stops a stolen ticket from turning into a man-in-the-middle.

The egress ticket is for the management cluster. From your cluster's point of view, the management cluster is just another client on the internet, and it still needs to reach your API server. The management cluster creates one ticket per cluster in memory, signs it with that cluster's own CA, and presents it only toward that cluster's endpoint. A ticket for cluster A is rejected by cluster B, because B does not trust A's CA. The management cluster reaches your API server over the load-balancer path, not the reverse tunnel.

The limits of the gate#

KubeGate gates connections that arrive from the internet. A caller already on the node skips it: a pod on the node's own network, or root over SSH, can reach the API server on its loopback address directly, without passing through the gate.

Skipping the gate grants no access. The API server still authenticates every request, so a caller on the node needs a valid identity like any other, and the gate never was the thing keeping them out. The only effect is on the audit log. A request that arrives on the node's loopback address is recorded as coming from the node itself, so it cannot be told apart from a control-plane component's own traffic. Cryptography cannot fix that, because the request genuinely originates on the node. The backstop is an alert on audit events that look like they come from the node's own loopback address.

Putting the API server behind KubeGate also removes the client's identity from the audit log. Every connection now arrives from the loopback address, so the log can no longer tell who the client was. A helper in the node agent restores the real client address into the audit log. It sets this locally on the node, after the connection has passed KubeGate, so a remote client cannot forge it.

The helper fails open. Reachability never waits on it, so if it is degraded the connection is served anyway and a marker is recorded instead. One marker means KubeGate handled the connection but could not name the caller.

IPv6 clients are the one case the helper cannot name at all. Publish only an IPv4 address for your control-plane endpoint, not an IPv6 one. The nodes use IPv4 on the loopback address, so an IPv6 client is served and recorded with a different marker. With IPv6 you keep reachability but lose the client's identity in the log, and Hetzner load balancers carry both IPv4 and IPv6 by default.

Keeping the gate working#

Two operational rules keep the gate doing its job. First, the address allowlist:

Warning

Two ways to break the address allowlist. Enable it only after PROXY protocol is live on the load balancer (the cluster reports it as enabled); before that KubeGate cannot see real client addresses and the check can be bypassed. And keep the trusted-address list to the load balancer's own address alone: an address listed both as trusted and inside an allowlist's range lets any client in that range choose the address the allowlist sees.

Second, remember what KubeGate is. It decides who may open a connection. It does not authenticate anyone; RBAC, the Kubernetes permission system, still decides what an admitted connection may do. once PROXY protocol is confirmed live, and alert on audit events attributed to the loopback address, because a legitimate client always leaves a real address or a marker. A plain loopback address with no marker means the connection never crossed the gate.

The services inside syself-agent#

syself-agent is built into the read-only image and covered by dm-verity like the rest of the disk. Each service runs as its own background process, so a failing one restarts by itself. These are the services it runs:

Service Runs on Job
Node agent every node keeps the local list of the cluster's nodes, runs the health checks that set node conditions, repairs what it can in place, and turns on the failover proxy
Tunnel agent every node dials the reverse tunnel out to each control plane and holds it open
Tunnel server control planes accepts tunnels and gives the local API server a path to every node
Failover proxy (syself-proxy) workers the kubelet failover proxy, reachable only from the worker itself

One job is not on this list: joining a node to the cluster. That runs outside syself-agent, as a separate first-boot program that exits once the node has joined.

The conditions the node agent sets are the signal the platform acts on to .

The reverse tunnel#

kubectl logs, exec, and port-forward all need the API server to reach a node's kubelet. The kubelet is the agent Kubernetes runs on each node to start and watch containers.

Instead of the API server dialing the node, the node dials out. Each node runs a tunnel agent that dials every control plane over mutual TLS (mTLS, where both sides prove their identity with certificates) and holds the connection open. A tunnel server on the control plane accepts the connection and gives the API server a local path into it. When the API server needs a node, its traffic goes back down the tunnel the node already opened.

The API server never dials a node, so nodes behind NAT or a strict firewall work unchanged.

sequenceDiagram
  participant API as API server
  participant S as Tunnel server (control plane)
  participant A as Tunnel agent (node)
  participant K as kubelet (node)

  Note over A,S: at boot, and held open from then on
  A->>S: dials out (mTLS, cluster CA)
  Note over API: later: kubectl logs for a pod on this node
  API->>S: node-bound request
  S->>A: sent down the tunnel the node opened
  A->>K: local connection to the kubelet only
  K-->>API: response returns the same way

The tunnel only reaches the local kubelet or an in-cluster address. It is not a general-purpose proxy.

The failover proxy#

A worker normally reaches the API server through one control-plane load balancer. Lose that load balancer for about forty seconds and the kubelet goes NotReady, Kubernetes moves the pods, and one load-balancer failure takes every worker down at once. The failover proxy, syself-proxy, removes that single point of failure.

It runs on the worker and is reachable only from that worker. It forwards kubelet traffic to the control planes and falls back to direct control-plane connections when the load balancer fails.

A check inside the node agent runs about every fifteen seconds. The check proves the full path through the proxy with the kubelet's own credentials, then points the kubelet at the local proxy and restarts it. The connection still verifies, because the API server's certificate also covers the loopback address.

Control planes skip the proxy, since their kubelet already talks to the local API server.

What listens on a node#

Every listener on a node falls into one of two groups: unreachable off the node, or reachable but authenticated. Nothing is reachable without authentication. KubeGate is the one public listener; it passes encrypted traffic through and the API server authenticates it. The tunnel server on control planes takes only cluster members, over mTLS behind the host firewall. Everything else, the API server, the failover proxy, and the tunnel's health and metrics, listens on the node itself and is unreachable from off the node. One more listener answers only the load balancer: a readiness check that reports whether the local API server is serving, so the load balancer sends traffic to a control plane only once it is ready.

If an attacker takes over one piece#

Take over any single reachable piece and you get only what that piece does:

Piece taken What it yields
KubeGate a raw-TLS forwarder; no credential, no CA key, no capabilities
a stolen node ticket the right to open a connection, for at most two hours; the API server still authenticates
a stolen egress ticket the same, and only against the one cluster whose CA signed it
the tunnel agent a route to the local kubelet on that node and to in-cluster addresses; every other connection is refused

How this adds up to a node being safe on the public internet is the , which needs .

How this connects to the two-cluster model#

KubeGate, the tunnel, and the tickets exist to connect the across the public internet. The guards the direct node-to-node path KubeGate never sees, and nodes use public addresses at all because the platform needs .

Next: .