How Syself Tests Every Cluster Stack Release

Syself tests every Cluster Stack release before you can use it. A release ships only when everything passes. We check that a cluster built from a release comes up and runs, that it recovers when parts of it break, and that every upgrade path between our own versions works. The heavy checks run on real clusters of different shapes, not on mocks or on reading the code.

What does Syself actually test?

A release is one fixed set of parts: the node operating system, the shape of the cluster, and hundreds of components that all have to agree with each other. The kernel, the container runtime, the kubelet, the network layer, the storage driver, the cloud integration, and more. Each one has its own version and its own rules about what it works with.

Change one part and it can break another several layers down. A network module built for one kernel version may not load on another. A container runtime the kubelet ran yesterday can be rejected by a newer kubelet tomorrow. These problems only surface when the parts run together on a real machine, and every cluster combines them a bit differently, so there are many combinations to work through. You prove them by building the combinations and running them, not by reading the code or trying a handful by hand.

From quick checks to full clusters

Before we release, we run the whole thing through several levels of testing, from quick file checks to full clusters under load. Each level catches a different kind of mistake.

• Fast checks, before any cluster exists. The quickest are gates that stop an easy mistake before a single machine is involved: a missing version, two components set to versions that disagree, a known security issue, a component past its end-of-life date. Next to them, unit-style tests confirm that our templates render the right configuration and that the logic behind them holds. They run on every change, so a broken combination is stopped before it ever reaches a cluster.
• Node-image checks, before we spend a machine. We check the finished node image two ways. Static checks confirm the boot settings are right, the container runtime is configured to match the kubelet, the images a node needs are already cached so it never has to pull at boot, the right services are enabled, the hardening is in place, and every piece of config lands where it belongs and is templated correctly. Then we boot the image in a virtual machine, feed it different inputs, and watch it come up, seal, and join before we spend a real machine on it.
• Real-cluster checks. We stand up live clusters and watch what they actually do: that every node reaches the Ready state, that the Kubernetes API answers, and that real workloads we deploy get scheduled, get an address, reach each other and the outside, serve traffic through a load balancer, keep their storage, and clean up when we delete them. This is the only level that proves a data path end to end. The tunnel, the pod network, storage, load balancing, and GPU workloads all get exercised on real machines.

The real-cluster level isn't one test but several kinds, each aimed at a different question:

• Creation — does a cluster built from this release come up and run?
• Configuration — do different inputs and cluster shapes all come up correctly, not just the default one?
• The upgrade journey — can a running cluster move to the next release, in order, without going down along the way?
• Self-healing — when parts of the cluster break, does it recover on its own?
• Workload behaviour — how does a running workload hold up under certain conditions, such as a node underneath it coming and going?

More than three hundred of these run on real clusters. Many more run without one, because those checks are cheap and quick. Building a Cluster Stack carries the weight of building anything people trust their lives to: every part has to be correct, and it has to be tested to prove it. So most of our engineering goes into the tests, not the product they guard, and much of that work covers the edge cases, the rare states that only surface when something breaks. When we hit a problem, we fix it for everyone and add a test that keeps it fixed.

A check only means something on a cluster that has the feature it checks: a GPU test needs a GPU node, an encryption test needs encryption turned on. So the release picks the cluster shape each check needs, and a check with no cluster to run on is marked not covered, never counted as a pass. A skipped check showing up green would hide the gap it was meant to find, so we check for that too.

Does it test failures, or just the happy path?

Both. A cluster behaving when nothing goes wrong is the easy case. The harder one is a cluster that keeps running when a node fails under real traffic, recovering on its own with no one stepping in.

So we break clusters on purpose. Our test scenarios drive a cluster through real events and watch how it reacts. One scenario kills a worker's kubelet and checks that the node heals itself, through one of three layers: the service restarts, the node reboots, or the platform replaces the machine. Exactly one of those has to fire, and a node that never recovers fails the test. Another scenario takes the control-plane load balancer out from under the workers and checks that they keep talking to the control plane anyway. We also remove nodes, cut the network, and drive upgrades while a workload is running, and check that the cluster does the right thing each time.

The full walkthrough of one of these paths is in control-plane failover.

How are upgrades tested?

Upgrades are the hardest part. An update and an upgrade are not the same thing:

• An update stays inside one Kubernetes minor track — say, moving to a newer 1-36 release. You get patches, CVE fixes, and OS and addon revisions. The Kubernetes API and the way the cluster behaves do not change.
• An upgrade moves to a new track — a new Kubernetes minor, like 1-36 to 1-37. This is where new features arrive, behaviour can change, and deprecated APIs can be removed.

Moving a live cluster in either direction is not one step. It has to happen in the right order, run its migrations, and keep the API server reachable the whole time. A cluster that goes down during its own upgrade is a failed upgrade, even if it comes back afterward.

You can't jump straight into a new minor from an old release. The path is to first catch up to the latest available release within your current track, then move to the latest available release of the next minor. That is exactly what we test: the upgrade from the latest release of the previous track into the new track. Which releases are available in each track, and how the tracks move forward, is on the Version Hub.

Why networking gets so much of the testing time

Networking is the part that is easy to get wrong and hard to keep right, so it takes the largest share of the real-cluster testing.

It is not just the network layer's own settings. It is how Kubernetes networking, the kernel modules, and the operating system fit together, and whether they keep matching under load. Those pieces move at different speeds, and a change in one can quietly break another. So a lot of a real-cluster test is spent proving they still line up: a pod gets an address, a Service routes to the right backend, the load balancer serves traffic and keeps the real client IP, and traffic that should be blocked from the outside is blocked.

How do you know you're running what we tested?

After a release passes, it's locked with a content hash. A cluster can only point at a name, and a name exists only after the tests for it pass, so a cluster can never run an untested combination. It's the same idea as an immutable node OS: the thing you run is fixed and identified, not assembled fresh each time.

Security rides along in the same process. Syself tracks CVEs and ships a vulnerability advisory with each release, in both OpenVEX and CSAF 2.0 formats, so a release is checked against known problems, not just built and shipped. Each release produces an SBOM with a verdict per CVE, a stack security report, and a recurring CVE feed with a patch deadline for each severity. The evidence is generated per release and published, so you can check it directly. Security evidence per release has the detail.

What we do not test: your workloads

The tests prove our stack. The operating system, the cluster shape, the components, and the upgrade path between our own parts. They do not prove that your workload is ready for its nodes to come and go.

In a Syself Autopilot cluster, nodes are replaced, not patched. That happens on an upgrade, but it also happens when a node fails and self-healing replaces it, and when a machine simply dies. In all three cases the effect on your workload is the same: a node it was running on goes away, its pods are drained, and they have to start again elsewhere. A workload that handles that loses nothing. A workload that does not will drop requests, whether the node left for an upgrade, a repair, or a crash.

So an upgrade does not create the problem. It surfaces one that was already there. If you fix it, you are covered for repairs and outages too. A few habits make a workload ready for a node to disappear:

• Run more than one replica of anything that matters, so losing one node never takes the last copy.
• Set a PodDisruptionBudget so a drain waits for a healthy replacement instead of taking too many pods down at once.
• Shut down gracefully so a pod finishes its in-flight requests when it is asked to stop.
• Use health probes so traffic only reaches a pod once it is actually ready.

Prepare workloads for upgrades walks through all of this in one place.

One thing applies only to an upgrade, that is, a move to a new Kubernetes minor, say 1-35 to 1-36. A new minor can remove a Kubernetes API your manifests still use. The stack upgrade succeeds, but your deployment breaks because the API it called is gone. An update within a track never does this: on the same minor, Kubernetes behaviour does not change and no API is removed, so your manifests keep working. No release test catches the cross-minor case either, because no release test reads your manifests. So before an upgrade to a new minor, run pluto and kubent against your own manifests to find those APIs. It takes a few minutes.

FAQ

Real clusters or simulations?

Both, and at different levels. We check code and configuration on their own first, where the parts have clear inputs and interfaces, so we can run a very large number of these cheaply. We boot the node image in a virtual machine and feed it different inputs to see how it behaves. And we run more than three hundred tests on real clusters with a real topology, with workloads running inside them, to see how the whole thing acts under load. A passing simulation is a good early signal. It is not proof a cluster works.

Do you test the exact upgrade I will run?

For an update within your track, every release is tested against the move from the one immediately before it, so going forward inside a track is a tested path. For an upgrade to a new minor, we test the move from the latest release of the previous track into the new track — so catch up to the latest release in your current track first, then upgrade. Done in that order, each step is a path we ran on real clusters. There is no supported way to skip a minor.

Do your tests cover my apps?

No. The tests prove our stack and the upgrade path between our own parts. Getting your workload ready for a node to come and go is your side of it. Run enough replicas, set a PodDisruptionBudget, shut down gracefully, and scan with pluto and kubent before an upgrade to a new minor.

The rule of thumb: trust the stack, get your workloads ready for a node to disappear, and to reach a new minor, catch up within your track first, then upgrade one minor at a time.