How We Secure Kubernetes Nodes Without a Private Network

Syself Autopilot manages your infrastructure as software. You describe the cluster you want, and the platform creates the servers, wires up the networking, heals what breaks, and keeps it all running on Hetzner. You stop thinking about machines and get to spend your time on Kubernetes and the workloads that pay the bills.

We've run Kubernetes on Hetzner in production since 2020. Over twenty engineers work on it now, specialists who maintain and contribute to the upstream Kubernetes projects and who build cluster-api-provider-hetzner, the open-source Cluster API provider for Hetzner. In 2023 we won the public tender for the Kubernetes-as-a-Service part of the Sovereign Cloud Stack, a German government initiative, with the best concept, and implemented the Cluster Stack approach there. Thousands of nodes later, run for agencies, enterprises, and startups, we've made it feasible to leave the big public clouds behind. At that scale, on real hardware, everything rests on a stable foundation. And that starts with keeping the nodes secure.

What a private network actually does

Almost everyone starts here, and we did too: put the nodes on a private network and they're hidden from the internet.

We looked deep into it. A private network buys you exactly one real benefit: a node with no public address at all can't be reached from the internet, so it stays out of range of anything scanning or probing from outside.

Most nodes don't get that benefit. A node that has both a public address and a private one is exactly as exposed as before: the private address changed nothing about the public door, which is still there and still answering.

It doesn't add isolation either. Traffic between your own servers already stays inside Hetzner's own network, even when it goes over public addresses. The private network gives your servers a second way to find each other on that same network; it doesn't wall anything off from anyone else.

And you always need at least one entry point. Somewhere there's a place you talk to your cluster, and that place is reachable. A private network doesn't remove that, it just moves it. So the work is protecting the entry point that has to exist either way.

The entry point is the Kubernetes API server

The entry point you actually talk to is the Kubernetes API server. Every kubectl command, every controller, every bit of automation goes through it, so that's where the protection has to be.

A firewall in front of it isn't enough on its own. The API server sits behind three checks: access control by policy, then authentication, then authorization.

It listens only on the node's loopback address, the address a machine uses to talk to itself, so nothing off the node reaches it directly. The only traffic we let reach a control-plane node's API port comes from the control-plane load balancer; nothing else on the internet can touch it. That traffic hits KubeGate, a small gateway we built that runs on the node in front of the API server, and KubeGate decides, by policy, whether the connection may open at all. You can narrow that to a set of trusted address ranges. KubeGate never decrypts anything: it passes the encrypted traffic straight through, and the API server presents its own certificate and authenticates every caller itself.

On top of those three checks, the audit log records the real client IP of each request, so every call traces to the source that made it. The mechanism has its own write-up in API server access control with KubeGate.

What makes it secure: layering

Because we run the infrastructure as software, we hold every node to the same standard, with no server left as a special case. Three layers work together, and the network topology isn't one of them.

A firewall that runs on the node itself

When people hear "firewall," they usually picture a separate box, or a service the cloud provider runs in front of your servers. Ours is neither. Every Syself node carries its own firewall and defends itself. There's no shared gate up front for the whole fleet, nothing in the path to route around or to fail on its own. It runs inside the Linux kernel using eBPF, a way to run small, safe programs in the kernel itself, so every packet is checked the moment it lands, before any program on the node can read it.

The default actions is to deny a connection. Nothing gets through unless it's on an explicit allow-list, and the list is keyed to who may connect, not to a port you happened to leave open. A scanner sweeping your address has its packets dropped in the kernel, and no process on the node ever receives them. Clearing the firewall still isn't getting in, as a packet that's allowed through still has to prove who it is.

Policy decides who may talk to what

Who's allowed to reach what is written down as policy and enforced. It is a decision we make and check, rather than a side effect of which network a machine landed on.

Identity, not location

We never trust a connection because of where it came from. There is no "inside." A packet from another node in the same cluster is treated like a packet from a stranger: it has to prove who it is first. That proof is mutual TLS, where both sides show a certificate, and the certificates rotate on their own. Only then is the request authenticated, and only then authorized. This is zero trust on the public network.

Firewall, policy, and identity together are what make the system secure. The layers don't lean on each other either: the firewall stays default-deny, the API server stays closed to the internet, and every connection still proves who it is, whatever else you turn on or off.

Where Hetzner's private network fits

If the private network were solid, we'd take it as one more layer on top of what we already have. More layers of protection are good. But two things make it less interesting than it sounds, in Hetzner's case.

First, a Hetzner private networks is not a VPC, as you may think. On the big clouds a VPC walls your network off from other tenants and lets you decide what may reach what, with rules the provider guarantees. Hetzner's private network isn't that. It hands your servers addresses on a shared network so they can find each other. No isolation, no encryption, so it can't be a security boundary on its own.

Second, it's unreliable at scale. It drops packets silently, so the failure doesn't show up until something built on top of it breaks, with no error to trace it back to. This is a well known issue, search and you'll find plenty of reports, on Reddit and elsewhere. It also has known issues on virtual switches and older hardware, and it's IPv4 only. There's a cap of around 100 servers too, though that's minor next to the reliability.

Hetzner is a great provider, so we use the good parts: we take servers and load balancers, and nothing else. The private network is a convenience for helping servers find each other, not a foundation we'd build security or scale on.

Proving who reached your cluster

Because access runs through our own gate and our own identity layer, we can name who made every connection and keep the record of it. That's what lets security-strict teams, such as those in enterprises and regulated industries, run here at all.

The real client address is carried through the gate, so the audit log names the actual caller instead of the load balancer in front of it. The kernel writes that address, so a client can't spoof it. Every node also runs an immutable audit ruleset that records privileged activity — logins, root commands, account and configuration changes, kernel-module loads — baked into the sealed image, so not even root can turn it off. The auditd ruleset reference lists exactly what's recorded, and Audit logs with the real client IP covers the client-IP enrichment.

We build in the technical controls that SOC 2, ISO 27001, and NIS2 require, so a cluster comes with what those frameworks ask for. What we ship is the evidence behind them: the audit streams, an SBOM, a vulnerability advisory, and a recurring CVE feed with each release. You operate the controls and make the case to your auditor; we hand you the raw material. Security evidence a release produces shows exactly what that is.

The controls you own

• SSH answers from anywhere until you restrict it. SSH listens on port 100, key-only, and it replies to any address until you add a deny policy. Lock it down to the ranges you use.
• Pods are default-allow until you segment them. The firewall and identity model protect the node and the control plane. Traffic between your own pods is open until you write a network policy, and no Pod Security level is set until you choose one.
• Pod-to-pod encryption between nodes is one switch away. Turn on WireGuard and pod traffic between nodes is encrypted; two pods on the same node never leave the machine, so that stays local. It's off by default because the traffic that matters most — the kubelet and the cluster database — already runs over mutual TLS. We'd recommend mTLS for your own workloads too, through a service mesh: it gives you attestation and finer control at L7, the application layer, on top of the encryption at the network layer.

FAQ

Isn't a public address on every node a bigger attack surface?

No. A public address isn't the same as an open door. The host firewall is default-deny, so a node answers only the identities on its allow-list, and a scanner's packets are dropped before any process reads them. The address is public; the ports are not open to the public.

Can I reach the API server directly over the internet?

Not directly. Port 6443 is closed to the internet except through the control-plane load balancer, and the API server itself listens only on loopback. Your traffic goes through KubeGate, which decides who may connect and then forwards to the API server, which authenticates you. You can narrow who gets through to a set of trusted ranges.

Would a private network make my cluster more compliant?

No. Compliance evidence comes from naming and recording every connection, which the gate and audit trail already do. A private network adds no audit signal, and it isn't part of how these clusters are built anyway.

Security isn't only the day you launch. We maintain Kubernetes minor versions over time, so you can stay current instead of falling behind, and the Version Hub shows the current state of the CVEs affecting your version. New ones keep surfacing, so we re-audit that state daily and you always have the full picture.

If you remember one rule: a connection is safe because it proved who it is, never because of where it came from. Protect the entry point, hold every node to the same standard in software, and the network topology stops mattering.