How We Secure Kubernetes Nodes Without a Private Network

Syself Autopilot manages your infrastructure as software. You describe the cluster you want, and the platform creates the servers, wires up the networking, heals what breaks, and keeps it all running on Hetzner. You stop thinking about machines and can spend your time on Kubernetes and the workloads your business runs on.

We've run Kubernetes on Hetzner in production since 2020. Over twenty engineers work on it now. They maintain and contribute to the upstream Kubernetes projects, and they build cluster-api-provider-hetzner, the open-source Cluster API provider for Hetzner. In 2023 we won the public tender for the Kubernetes-as-a-Service part of the Sovereign Cloud Stack, a German government initiative, with the best concept. We implemented the Cluster Stack approach there. We've since run thousands of nodes for agencies, enterprises, and startups, and made it practical for them to leave the big public clouds. At that scale, on real hardware, everything depends on a stable foundation, starting with secure nodes.

What a private network actually does 

Almost everyone starts here, and we did too: put the nodes on a private network and they're hidden from the internet.

We looked into it closely. A private network gives you exactly one real benefit: a node with no public address at all can't be reached from the internet, so it stays out of range of anything scanning or probing from outside.

Most nodes don't get that benefit. A node that has both a public address and a private one is exactly as exposed as before: adding a private address changes nothing about the public one, which is still there and still answering.

It doesn't add isolation either. Traffic between your own servers already stays inside Hetzner's own network, even when it goes over public addresses. The private network gives your servers a second way to find each other on that same network. It doesn't isolate them from anyone else.

You always need at least one entry point. Somewhere there's a place you talk to your cluster, and that place is reachable. A private network only moves that entry point somewhere else, so either way the job is to protect it.

The entry point is the Kubernetes API server 

The entry point you actually talk to is the Kubernetes API server. Every kubectl command, every controller, every bit of automation goes through it, so that's where the protection has to be.

A firewall in front of it isn't enough on its own. The API server sits behind three checks: access control by policy, then authentication, then authorization.

It listens only on the node's loopback address, the address a machine uses to talk to itself, so nothing off the node reaches it directly. The only traffic we let reach a control-plane node's API port comes from the control-plane load balancer. Nothing else on the internet can reach it. That traffic hits KubeGate, a small gateway we built that runs on the node in front of the API server, and KubeGate decides, by policy, whether the connection may open at all. You can narrow that to a set of trusted address ranges. KubeGate never decrypts anything. It passes the encrypted traffic straight through, and the API server presents its own certificate and authenticates every caller itself.

On top of those three checks, the audit log records the real client IP of each request, so every call traces to the source that made it. The mechanism has its own write-up in API server access control with KubeGate.

What makes it secure: layering 

Because we run the infrastructure as software, we hold every node to the same standard, with no server left as a special case. Three layers work together, and the network topology isn't one of them.

A firewall that runs on the node itself 

When people hear "firewall," they usually picture a separate box, or a service the cloud provider runs in front of your servers. Ours is neither. Every Syself node runs its own firewall. There's no shared gate in front of the fleet that could be routed around or fail on its own. It runs inside the Linux kernel using eBPF, a way to run small, safe programs in the kernel itself, so every packet is checked the moment it lands, before any program on the node can read it.

The default action is to deny a connection. Nothing gets through unless it's on an explicit allow-list, and the list is based on who may connect rather than on which ports happen to be open. A scanner sweeping your address has its packets dropped in the kernel, and no process on the node ever receives them. Getting past the firewall isn't enough to get in, because a packet that's allowed through still has to prove who it is.

Policy decides who may talk to what 

Who's allowed to reach what is written down as policy and enforced. Access is a deliberate decision we make and check. It doesn't depend on which network a machine happens to be on.

Identity, not location 

We never trust a connection because of where it came from, and no part of the network counts as "inside." A packet from another node in the same cluster is treated like a packet from a stranger: it has to prove who it is first. That proof is mutual TLS, where both sides show a certificate, and the certificates rotate on their own. Only then is the request authenticated, and only then authorized. This is zero trust on the public network.

Together, the firewall, policy, and identity make the system secure. The layers also don't depend on each other: the firewall stays default-deny, the API server stays closed to the internet, and every connection still proves who it is, whatever else you turn on or off.

Where Hetzner's private network fits 

If the private network were solid, we'd take it as one more layer on top of what we already have. More layers of protection are good. In Hetzner's case, two things make it less useful than it sounds.

First, a Hetzner private network isn't a VPC, even if it looks like one. On the big clouds a VPC walls your network off from other tenants and lets you decide what may reach what, with rules the provider guarantees. Hetzner's private network isn't that. It hands your servers addresses on a shared network so they can find each other. It has no isolation and no encryption, so it can't be a security boundary on its own.

Second, it's unreliable at scale. It drops packets silently, so the failure doesn't show up until something built on top of it breaks, with no error to trace it back to. It's a well-known issue, with plenty of reports on Reddit and elsewhere. It also has known issues on virtual switches and older hardware, and it's IPv4 only. There's also a cap of around 100 servers, though that matters less than the reliability problems.

Hetzner is a great provider, and we use the parts we rely on: servers and load balancers, and nothing else. The private network is a convenience for helping servers find each other, and we wouldn't build security or scale on it.

Proving who reached your cluster 

Because access runs through our own gate and our own identity layer, we can name who made every connection and keep a record of it. That's what lets teams with strict security requirements, such as enterprises and regulated industries, run here.

The real client address is carried through the gate, so the audit log names the actual caller instead of the load balancer in front of it. The kernel writes that address, so a client can't spoof it. Every node also runs an immutable audit ruleset that records privileged activity: logins, root commands, account and configuration changes, and kernel-module loads. The ruleset is baked into the sealed image, so not even root can turn it off. The auditd ruleset reference lists exactly what's recorded, and Audit logs with the real client IP covers the client-IP enrichment.

We build in the technical controls that SOC 2, ISO 27001, and NIS2 require, so a cluster comes with what those frameworks ask for. What we ship is the evidence behind them: the audit streams, an SBOM, a vulnerability advisory, and a recurring CVE feed with each release. You operate the controls and make the case to your auditor. Security evidence a release produces shows exactly what that is.

The controls you own 

• SSH answers from anywhere until you restrict it. SSH listens on port 100, key-only, and it replies to any address until you add a deny policy. Lock it down to the ranges you use.
• Pods are default-allow until you segment them. The firewall and identity model protect the node and the control plane. Traffic between your own pods is open until you write a network policy, and no Pod Security level is set until you choose one.
• Pod-to-pod encryption between nodes is one switch away. Turn on WireGuard and pod traffic between nodes is encrypted. Traffic between two pods on the same node never leaves the machine, so it stays local. It's off by default because the traffic that matters most, from the kubelet and the cluster database, already runs over mutual TLS. We'd recommend mTLS for your own workloads too, through a service mesh: it gives you attestation and finer control at L7, the application layer, on top of the encryption at the network layer.

FAQ 

Isn't a public address on every node a bigger attack surface? 

No. Having a public address doesn't mean the node is open. The host firewall is default-deny, so a node answers only the identities on its allow-list, and a scanner's packets are dropped before any process reads them. The address is public, but its ports aren't open to the public.

Can I reach the API server directly over the internet? 

Not directly. Port 6443 is closed to the internet except through the control-plane load balancer, and the API server itself listens only on loopback. Your traffic goes through KubeGate, which decides who may connect and then forwards to the API server, which authenticates you. You can narrow who gets through to a set of trusted ranges.

Would a private network make my cluster more compliant? 

No. Compliance evidence comes from naming and recording every connection, which the gate and audit trail already do. A private network adds no audit signal, and it isn't part of how these clusters are built anyway.

Security work continues after launch. We maintain Kubernetes minor versions over time, so you can stay current instead of falling behind, and the Version Hub shows the current state of the CVEs affecting your version. New ones keep surfacing, so we re-audit that state daily to keep it current.

To sum up, we trust a connection because it proved who it is, never because of where it came from. We protect the entry point and hold every node to the same standard in software, so security doesn't depend on the network topology.