Performance of local vs network storage
Put a database on a server with a local NVMe SSD, and it runs faster than the same database in a virtual machine whose storage is a network-attached array. This is not a platform feature or a Kubernetes trick. It is physics, and it holds on any infrastructure, in any cloud.
A local NVMe drive connects to the CPU directly over the machine's PCIe bus. A read or write reaches it and returns in microseconds, with no network in the way. Network-attached storage puts a longer road in between: the request leaves the machine through a network card, crosses a switch, reaches a separate storage system, waits on that system's controllers, and travels all the way back before it completes. Every operation pays for that trip. A drive that skips it cannot help but be faster, and a database issues a steady stream of small operations, so the gap compounds across all of them.
Fewer parts also means fewer things that can break. The local path is a CPU and a disk. The network-attached path adds cards, cables, switches, and a storage array, and each one is a component that can fail or stall. For a database, local storage is faster and simpler at the same time.
On Syself Autopilot this is a bare metal server with its own NVMe, served to Kubernetes through TopoLVM. The platform makes the disk easy to use; the speed comes from the hardware.
Where the time goes#
The platform gives you three storage paths, and each is a different distance from the CPU. A Hetzner Cloud volume lives on separate storage hardware, so it pays that network trip on every read and write. A shared filesystem like JuiceFS pays more still: it reads from an object store and consults a separate metadata service, so a single file operation can turn into several round trips. Local disk pays neither cost.
The ordering falls out of that:
- Local NVMe has the lowest latency and the highest IOPS, because there is no hop and NVMe is the fastest drive class.
- Network-attached cloud volumes add a network round trip to every operation, so latency is higher and throughput is capped by the network path. That hop buys durability: Hetzner stores each volume on Ceph with three copies and reattaches it to a new node, so it survives a disk failure and follows the workload, which a local disk does not.
- A shared filesystem such as JuiceFS sits behind an object store plus a metadata lookup, so its latency is higher again and varies more from one operation to the next.
| Option | Latency character | IOPS character | Redundancy | Follows a moved workload? | Best for |
|---|---|---|---|---|---|
| Local NVMe | Lowest, tight and steady | Highest | None, one copy on one server | No | Latency-sensitive databases |
| Hetzner Cloud volume | Higher, one network round trip | Capped by the network path | Ceph, three copies | Yes | Data that must follow the workload |
| JuiceFS | Highest and most variable | Lowest of the three | From the object store | Yes | Many writers sharing one filesystem |
Note
The comparison here is about ordering and character, not fixed numbers. Real latency and IOPS depend on the drive model, the request size, the queue depth, and how busy the server is. Measure on your own hardware before you quote a figure.
Predictable, not just fast#
Low latency matters less than predictable latency for most stateful workloads. A local NVMe drive serves one server's workloads, so its response time barely moves from one request to the next. You get a tight, steady latency band. Network and shared storage draw from capacity that other tenants and other workloads share, so a busy neighbor can slow your operations at the worst moment. The average might look fine while the tail, the slowest few percent of requests, is what actually hurts a database under load.
A database issues many small, latency-sensitive operations: commits to the write-ahead log, index lookups, fsync on commit. A long, unpredictable tail on those turns steady query times into erratic ones.
Local NVMe keeps every database operation off the network and out of any storage translation layer, which is the latency a database actually feels. A generic replicated volume hands that advantage back, because the extra layer adds latency of its own. For the full case, that a production database belongs under an operator such as CloudNativePG rather than on a generic replicated volume, see Choose storage for a workload .
When the network hop does not matter#
Not every workload feels the difference. The network hop is only a cost when your workload is sensitive to it.
- Large sequential reads and writes, like streaming a big file, are limited by throughput, not per-operation latency. A network volume with enough bandwidth keeps up.
- Workloads that cache in memory and rarely touch the disk spend little time waiting on storage either way.
- Data that must follow the workload across servers needs network or shared storage regardless of speed, because local disk cannot move with the pod. Choose storage for a workload weighs that tradeoff.
Reach for local disk when small-operation latency and its predictability drive your workload, and give it a backup, plus application-level replication, to cover its single copy. When durability, mobility, or many writers matter more than the last microsecond, network or shared storage is the right call.
Benchmark with your real access pattern#
A synthetic benchmark rarely predicts your workload, so measure with the access pattern you actually run. A database doing small random reads and syncing writes on commit stresses storage in a completely different way than a job streaming large files. Run the real workload, or a load test that mirrors its request sizes and its mix of reads and writes, and watch the latency tail, not only the average.
Match the disk type to that pattern too. The local classes are not interchangeable:
local-nvmefor the most demanding databases and any latency-sensitive small-operation workload.local-ssdfor solid random performance at more capacity per server, where NVMe would be more than the workload needs.local-hddfor large, mostly sequential data where capacity per terabyte matters more than latency.
When you are ready, set these classes up on your servers with set up local NVMe with TopoLVM , then tune storage performance to move a running workload's numbers.
Data survival across reprovision
Syself Autopilot rewrites the OS disk on every reprovision and never touches your data disks, so local data survives in place unless the workload moves to another server.
ReadWriteMany with JuiceFS
Stand up a shared ReadWriteMany filesystem on Syself Autopilot with JuiceFS, backed by S3-compatible object storage and a Redis or PostgreSQL metadata engine.