Skip to main content

Tune storage performance

Inspect 1.36

Faster storage usually means using the disk you already have better, not buying a bigger one. Most of the gains come from matching the workload to the right class, laying down the right filesystem, sizing the thin pool sensibly, and making sure the pod is not being throttled before the disk is even the limit. Work the levers in that order.

Lever When it helps What to do
Right class Latency-sensitive, random I/O workloads Put databases and queues on local-nvme; keep cold or streaming data on local-hdd or cloud volumes
Filesystem Heavy parallel writes and very large files Stay on ext4 by default; choose xfs for high concurrency
Thin-pool sizing Many small volumes versus big sequential files Set chunk and metadata sizing at lvcreate time, with metadata headroom
JuiceFS cache Read-heavy workloads on the juicefs class Put the cache on NVMe and size it to the working set the pods reread
Resource requests A pod throttled before the disk is the limit Set honest CPU and memory requests, and check throttling metrics first
Noisy neighbor Mixed I/O profiles sharing one pool or disk Separate them by node type or pool, and cap each pod with resource requests

Start with the class#

The largest single factor is which the volume comes from, because that decides what physical disk is underneath. A workload that does many small random reads and writes, a database or a message queue, wants local-nvme, where the disk is inside the server and there is no network in the path. A workload that streams large files or keeps cold data is fine on local-hdd or on cloud volumes. Putting a latency-sensitive database on the wrong class is a ceiling no amount of later tuning lifts. explains why the gap is so wide.

Filesystem: ext4 or xfs#

Both ext4 and xfs are solid, and the difference between them is smaller than the difference between disk types. Pick on the shape of the workload rather than a blanket rule:

  • ext4 is the safe default. It is predictable, well understood, and fine for the large majority of workloads.
  • xfs tends to hold up better under heavy parallel writes and very large files, because it was built for high concurrency.

If you have no strong reason either way, stay on the default and spend the effort on the class and the pod instead.

Size the thin pool#

A local class carves volumes from an LVM thin pool, and the pool has two sizing knobs set when you create it. The chunk size is the unit the pool allocates in. A larger chunk lowers metadata overhead and suits big sequential files; a smaller chunk wastes less space on many small volumes but uses more metadata. The metadata area itself has to be large enough that it does not fill before the data does, since exhausting metadata freezes the pool exactly like running out of data space.

Tip

Decide chunk and metadata sizing when you first create the pool with lvcreate. The chunk size cannot be changed later, so pick with the workload in mind up front and leave metadata headroom.

Tune the JuiceFS cache#

For the juicefs class the data lives in object storage, so the local cache is what stands between a pod and a network round trip on every read. A larger local cache on fast disk keeps hot data close and cuts latency sharply for read-heavy workloads. Put that cache on NVMe rather than a slow disk, and size it to the working set the pods actually reread. covers setting up the class.

Do not let the pod throttle itself#

Storage that looks slow is sometimes a pod starved of CPU or memory. Encryption, compression, and the CSI driver all spend CPU, and a pod pinned under a tight CPU limit stalls before the disk is the bottleneck. Set honest resource requests so the scheduler gives the pod real headroom, and check CPU throttling metrics before blaming the disk. On a disk with , the cipher runs on the node CPU, so this matters more.

Watch for noisy neighbors#

Volumes that share one pool or one disk also share its throughput. A batch job hammering the disk can drown out a database next to it, even though each has its own volume. Keep workloads with very different I/O profiles on separate node types or separate pools, and use resource requests to stop one pod from monopolizing a shared disk.

Measure before and after#

Change one thing at a time and measure the effect on your own hardware. Benchmark numbers from someone else's setup do not carry over, because the disk model, the workload, and the encryption choice all move the result. Capture a baseline, make one change, and compare.

Related: for the per-volume and per-pool metrics that tell you whether a change actually helped.