Every server running your workloads is locked down in layers, each targeting a different attack surface: a sealed OS that cannot change at runtime, a hardened kernel, a default-deny firewall, key-only SSH, and a BSI C5 audit trail. [Syself Linux](/docs/hetzner/apalla/concepts/internals/syself-linux) covers the overall model for the OS. ## A sealed OS that cannot change Syself Linux is read-only and immutable. It is around 350 MB, built once, and never patched: when it has to change, the node is replaced from a fresh image, not updated in place. See [the node lifecycle](/docs/hetzner/apalla/concepts/operations/node-lifecycle). There is no package manager, and nothing installs or changes at runtime, so nothing on a running node drifts from what Syself shipped. dm-verity enforces that. The kernel splits each sealed image into blocks, hashes them into one root hash fixed at build time, and rechecks every block against that hash on every read. A modified block fails the read, so a tampered layer cannot be used. A node runs from sealed layers, each its own dm-verity image. The node OS is a sealed base plus the extra components a node's role or hardware needs. The base holds the kernel, the kubelet, the container runtime, and systemd. On top of it sit control-plane tooling on control-plane nodes, the NVIDIA driver on GPU machines, and the `secure` runtime, each verified. The per-node config image holds this node's hostname, machine identity, SSH host keys, and network and kubelet configuration, one per node. The base's root hash sits on the kernel command line, so the kernel checks it from the first read at boot. Each add-on layer's hash is fixed at build time the same way. The per-node config image is sealed on the node during provisioning, so its hash cannot be fixed at build time; the provisioner records it on the node. Every read from any sealed layer is checked from then on. A layer that does not match its recorded hash cannot be read. The partitions never change: | Partition | Contents | Writable? | | --------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------- | | `EFI` | The bootloader, the kernel, and the boot line | Yes. This is the trust boundary (see Limits). | | `OS` | The sealed base and add-on layers the kernel verifies with dm-verity | No. Verified on every block read. | | `HASH` | The dm-verity hash trees for the sealed OS layers | Never mounted; the kernel reads it directly. | | `BIOS` | The bootloader core for BIOS boot | No. | | `DATA` | Everything writable: certificates, kubelet state, image cache, logs, and the per-node config image | Yes. Tamper-evident, not verified. | Containers run on crun by default, with runc as a fallback. A workload that needs stronger isolation than a shared kernel can run in its own lightweight VM by setting `runtimeClassName: secure` in the pod spec, on bare-metal nodes with hardware virtualization. ## A kernel that cannot drift Kernel hardening is fixed at build time and verifiable on the node. The configuration states what the kernel must have on, what must be built in rather than loadable, and which driver families are deliberately off. It cannot silently regress between releases. Driver families no Hetzner server has (Bluetooth, Wi-Fi, dozens of legacy filesystems and protocols) are compiled out at the top level and stay out. Code that is not in the kernel cannot be exploited. You can see what a release contains and how it was built. Every component is pinned to a content hash at its origin and fetched fail-closed against that hash, so a swapped or tampered source stops the build instead of reaching a node. For this release, the [Version Hub](/versions) lists every component with its CVE and SBOM evidence, and each release ships a machine-readable SBOM you can hand to a scanner or an auditor; see [Security evidence a release produces](/docs/hetzner/apalla/security/security-evidence-per-release) and [Supply-chain security](/docs/hetzner/apalla/security/supply-chain-security). On any node, the running kernel exposes its full configuration, so `zcat /proc/config.gz` shows you exactly what was built. One caveat: the kernel's lockdown security module is compiled in but inactive by default. ## Module signing with no exceptions Every kernel module in the image is signed with SHA-512 at build time, and the kernel refuses to load an unsigned module (`MODULE_SIG_FORCE`). There are no carve-outs. The out-of-tree NVIDIA modules, both driver branches, are compiled against the same kernel tree and signed with the same key as everything else. A classic loadable-kernel-module rootkit has to load an unsigned module, and it cannot: the kernel loads none. The signing key is generated inside that image's build and discarded with it. Nobody holds the key after the build, so it cannot be stolen or reused against a running fleet. The trade-off: an independent rebuild from the same source signs with its own key and produces different bits. Reproducibility on Syself Linux means the base image's root hash can be re-derived from the released image with standard tools, not that a rebuild yields identical bits. A note on `kexec`, the mechanism for booting a new kernel without a reboot: the legacy system call that loads unsigned kernel images is removed, and the remaining file-based path verifies a signature when one is present. It does not hard-refuse an unsigned image, because signature enforcement for kexec is not switched on and lockdown is inactive. ## A default-deny network Every node runs a host firewall in default-deny mode. Traffic not on an explicit allow-list is dropped before it reaches any process. Kubernetes internals (the kubelet, cluster networking, and etcd) are reachable only from within the cluster. The Kubernetes API server sits behind the cluster's control-plane load balancer and is not directly reachable from the internet. SSH on port 100 is the only port a node accepts from the outside. For the complete port list, see [Ports and listeners](/docs/hetzner/apalla/reference/ports-and-listeners). The firewall keys its allow-rules to identity, not to a trusted network, which is why nodes run safely on the public internet with no private network. [Zero trust on public networks](/docs/hetzner/apalla/security/zero-trust-on-public-networks) covers how every connection into a node authenticates. Pod-to-pod traffic is a separate layer you own; it starts default-allow until you segment it with [network policies](/docs/hetzner/apalla/security/segment-with-network-policies). Nodes are IPv4-only. A service that binds to all interfaces cannot be reached via IPv6 because the IPv6 stack is not active on the node. IPv6 terminates at the load balancers. ## Key-only SSH SSH listens on port 100, not the standard port 22. Password login is disabled for all users. Every login requires a key. There is no root password set. The SSH configuration is sealed in the OS and cannot be changed on a running node. Every command run in a root SSH session is recorded in the audit log. SSH is on by default so support can always reach a node. It is the only way in that does not depend on Kubernetes: if the cluster API is down, SSH is how support still gets in. You can restrict it to your own network or block it completely, see [Restrict SSH access](/docs/hetzner/apalla/security/restrict-ssh-access). If you block it, be aware that support and debugging are then limited to what the cluster API and the log collector can reach. ## A complete audit trail Every administrative action on a node is recorded. Syself Linux ships with an audit ruleset written for BSI C5:2020, the German Federal Office for Information Security's cloud security catalogue. Its logging and monitoring requirements call for an audit trail of changes to identity, access, and system configuration. The ruleset records: - Changes to the audit configuration itself, and deletion of audit log files by a login session (deletion by a process with no login identity, such as a daemon or a container process, is not recorded) - Changes to user accounts, SSH keys, and privilege files - Changes to Kubernetes and container runtime configuration - Every command run in a login session, including break-glass root access - Kernel module loads and unloads - Clock changes - Process-tracing and mount calls from a login session The ruleset closes its own evasions. Session rules are scoped to the login identity the kernel stamps on a session (the auid), and by default a root process can reset its own login identity and drop out of every one of those rules. The ruleset pins the login identity so that write is final (`--loginuid-immutable`), then locks all audit configuration until the next reboot (`-e 2`). Even with root access you cannot clear or reshape the rules on a running node. The rule files ride the sealed OS layer, so changing them takes a modified layer and a reboot, and the modification shows up as a tamper signal. Local logs are tamper-evident, not tamper-proof. The rules record deletion of the audit log files by a login session, but a root process that never went through a login carries no login identity, so it can delete them unrecorded. Logs are stored locally with about 400 MB of capacity. Continuous off-node shipping to a long-term, append-only store is the real control here. See [Retrieve audit logs](/docs/hetzner/apalla/security/retrieve-audit-logs) and the [Auditd rules reference](/docs/hetzner/apalla/reference/auditd-rules). > [!NOTE] > There is a small gap between the start of the boot process and when the audit daemon takes over. Events in this early-boot window are not covered. ## Tamper detection at runtime The OS layers are immutable. The writable partition where runtime data lives cannot be made immutable, so the platform watches it instead. Every three minutes the node monitor re-checks a baseline of protected files: Kubernetes CA certificates, signing keys, static pod manifests, cluster configuration, and an executable scan of the writable partition, excluding the high-churn runtime directories (containerd and its image cache, etcd, kubelet state, the CNI binary directory, the per-node config image, logs, cache, tmp and spool) where normal operation writes constantly. When it finds an unexpected addition, removal, or modification, it raises a `NodeTampered` condition on that node and publishes the finding to the cluster's API server, so the record lives in the control plane's etcd and not only on the affected node's disk. Consider what an intruder on the node would have to do to hide the change. Reverting the change before the next check does not work: the condition latches, so restoring the file does not clear it. Wiping the node's disk does not erase it either: the condition and the report are already stored in the cluster's API, not only on the node. They are not beyond the node's reach, though. The daemon writes them with the node's own kubelet credential, so root on that node can clear them too, and on a control-plane node the etcd holding them runs on the node itself. Ship the condition transitions and the report annotation to an append-only off-cluster store if you need evidence a compromised node cannot touch. One gap remains: the window before the first check runs, the baseline limit described under Limits below. Detection and response are split on purpose. On cloud nodes, a block-read failure in a sealed OS layer (the `VerityCorruption` condition) replaces the machine automatically after 60 seconds: laying the image down again fixes a corrupt image, and waiting only keeps a node with an unverifiable OS serving traffic. `SealedOSTampered` and `NodeTampered` alert instead of replacing, so the node and its evidence survive for investigation. On bare metal, even verity corruption goes to a human, because a failing read is as likely a failing disk as tampering, and re-provisioning the same physical server would churn the pool without fixing it. See [Node health conditions](/docs/hetzner/apalla/reference/node-health-conditions) and [Verify node integrity](/docs/hetzner/apalla/security/verify-node-integrity) for what each condition means and how to act on it. When a condition fires, [Respond to a tampered node](/docs/hetzner/apalla/security/respond-to-a-tampered-node) is the runbook: preserve evidence, rotate what the node held, then reprovision. ## What you can verify yourself You can confirm that the OS running on a node matches what Syself built, without reading the source code. Check a node's live dm-verity root hash against the value Syself published for the release. The build is reproducible at the image level, so you recompute the hash from the released image with stock tools and compare. [Verify node integrity](/docs/hetzner/apalla/security/verify-node-integrity) is the walkthrough. Every release also ships the supply-chain records an auditor asks for: the SBOM, the vulnerability advisory, and the recurring CVE feed. [Security evidence a release produces](/docs/hetzner/apalla/security/security-evidence-per-release) covers what each artifact is and how to read it. **The OS protection is software-rooted, not hardware-rooted.** Sealed means the base, its add-on layers, and the per-node config image are each a dm-verity image, verified block by block by the kernel at runtime. The sealing is keyless: the kernel checks image data against a root hash, not a signature over it, and no signing key ships in the image. Integrity is the kernel's dm-verity check on every layer, and authenticity is content-hash pinning plus Syself's provisioner and control plane knowing what was built and deployed. It does not use the server hardware security chip (the TPM) to seal keys, to measure the boot chain, or to prove the node's state to a remote checker (remote attestation), and it does not use Secure Boot. Those capabilities are out of scope, not planned. The trust boundary is the boot line on the writable EFI partition: whoever can rewrite it chooses what the node boots, and everything below it is enforced by the kernel. This is a real root of trust for "has this OS been modified?" but the trust is grounded in software, not in the server's hardware security chip. **The tamper baseline covers only what is there when the monitor starts.** A change made while the monitor was offline is adopted into the new baseline on restart and is not flagged. Anchoring the baseline to a sealed manifest at build time is planned. **There is no built-in long-term store for audit events.** Push the audit log stream off-node continuously to a SIEM or append-only store.