ReadWriteMany with JuiceFS
A shared filesystem is the fallback for an app that cannot use object storage, so confirm you need one first with Choose an RWX backend .
JuiceFS gives you a ReadWriteMany filesystem that pods on different nodes mount read-write at the same time. It stores file data in object storage and file metadata in a database, so the data lives independently of your nodes. That fits Syself Autopilot well: when a node is replaced, there is no volume tied to it to recover.
Reach for JuiceFS when an application needs a shared filesystem and cannot use the S3 API directly. If the app can speak S3, object storage itself is the simpler choice; JuiceFS puts a filesystem in front of that same object storage for apps that need one.
JuiceFS is yours to run. The node images ship with FUSE support, which JuiceFS needs to mount the filesystem, so there is nothing to enable on the nodes. You provide the object store, the metadata engine, and the CSI driver. If you would rather not operate the object store and metadata engine yourself, the Syself Consulting team can set JuiceFS up with you or run it as a full service.
What you need first
- Object storage for the file data. Run SeaweedFS in the cluster, or use any S3-compatible bucket. You need the endpoint, a bucket, and an access key and secret.
- A metadata engine. JuiceFS keeps the directory tree and file attributes in a fast database. Redis or PostgreSQL both work. This engine is on the write path for every file operation, so run it with the availability your workload needs. If it is down, the filesystem stalls.
Install the CSI driver
Deploy the JuiceFS CSI driver with Helm:
$ helm repo add juicefs https://juicedata.github.io/charts/
$ helm repo update
$ helm upgrade --install juicefs-csi-driver juicefs/juicefs-csi-driver \
--namespace kube-system \
--set node.ifPollingKubelet=false \
--set immutable=true
Warning
If you installed the CSI driver before 1.36, re-run the command above when you upgrade. node.ifPollingKubelet and immutable are both new requirements: pods can no longer reach the kubelet on port 10250, and the node filesystem is read-only. Without them the driver crashloops and the mount pods never start.
Format the filesystem once
Formatting writes the filesystem's settings into the metadata engine. Do it once, from anywhere the juicefs CLI can reach both the object store and the metadata engine:
$ juicefs format \
--storage s3 \
--bucket https://<s3-endpoint>/<bucket> \
--access-key <access-key> \
--secret-key <secret-key> \
redis://:<password>@<redis-host>:6379/1 \
myjfs
The last two arguments are the metadata URL (here a Redis database) and the filesystem name (myjfs). For PostgreSQL, pass a postgres:// URL instead.
Wire it into Kubernetes
Store the same connection details in a Secret the driver reads:
apiVersion: v1
kind: Secret
metadata:
name: juicefs-sc-secret
namespace: kube-system
type: Opaque
stringData:
name: myjfs
metaurl: redis://:<password>@<redis-host>:6379/1
storage: s3
bucket: https://<s3-endpoint>/<bucket>
access-key: <access-key>
secret-key: <secret-key>
Create a StorageClass that points at it. Object-backed volumes bind immediately, because there is no node or location to wait for:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs
provisioner: csi.juicefs.com
volumeBindingMode: Immediate
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
$ kubectl apply -f juicefs-secret.yaml -f juicefs-sc.yaml
Create an RWX claim and prove it works
Request ReadWriteMany on the juicefs class:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-data
spec:
accessModes:
- ReadWriteMany
storageClassName: juicefs
resources:
requests:
storage: 10Gi
Schedule a writer and a reader on different nodes, both mounting the same claim. The anti-affinity rule is what keeps them on separate nodes:
apiVersion: v1
kind: Pod
metadata:
name: writer
labels:
app: juicefs-check
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {app: juicefs-check}
topologyKey: kubernetes.io/hostname
containers:
- name: app
image: nginx
volumeMounts:
- mountPath: /data
name: shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-data
---
apiVersion: v1
kind: Pod
metadata:
name: reader
labels:
app: juicefs-check
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {app: juicefs-check}
topologyKey: kubernetes.io/hostname
containers:
- name: app
image: nginx
volumeMounts:
- mountPath: /data
name: shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-data
$ kubectl apply -f writer-reader.yaml
Once both pods are Running, confirm the reader sees the writer's file:
$ kubectl exec writer -- sh -c 'echo "shared across nodes" > /data/hello.txt'
$ kubectl exec reader -- cat /data/hello.txt
shared across nodes
Note
The driver runs one mount pod per PVC per node. Several pods on the same node that use the same claim share a single mount; a pod on another node gets its own. Expect a mount pod to appear in kube-system for each node that touches the volume.
Related: Shared storage with NFS for when NFS is the simpler choice ยท Choose an RWX backend to decide whether you need ReadWriteMany at all.
Performance of local vs network storage
Local disks skip the network hop, so latency stays low and predictable, which is why databases belong on them instead of network-attached storage.
Shared storage with NFS
Serve a ReadWriteMany filesystem on Syself Autopilot with NFS, using an in-cluster server or the NFS subdir provisioner for dynamic PVCs.