ReadWriteMany with JuiceFS

A shared filesystem is the fallback for an app that cannot use object storage, so confirm you need one first with Choose an RWX backend .

JuiceFS gives you a ReadWriteMany filesystem that pods on different nodes write at the same time. It stores file data in object storage and file metadata in a database, so the data lives independently of your nodes. That fits Syself Autopilot well: when a node is replaced, there is no volume tied to it to recover.

Reach for JuiceFS when an application needs a shared filesystem and cannot use the S3 API directly. If the app can speak S3, object storage itself is the simpler choice; JuiceFS puts a filesystem in front of that same object storage for apps that need one.

JuiceFS is yours to run. The node images ship with FUSE support, which JuiceFS needs to mount the filesystem, so there is nothing to enable on the nodes. You provide the object store, the metadata engine, and the CSI driver. If you would rather not operate the object store and metadata engine yourself, the Syself Consulting team can set JuiceFS up with you or run it as a full service.

What you need first#

• Object storage for the file data. Run SeaweedFS in the cluster, or use any S3-compatible bucket. You need the endpoint, a bucket, and an access key and secret.
• A metadata engine. JuiceFS keeps the directory tree and file attributes in a fast database. Redis or PostgreSQL both work. This engine is on the write path for every file operation, so run it with the availability your workload needs. If it is down, the filesystem stalls.

		$ helm repo add juicefs https://juicedata.github.io/charts/
$ helm repo update
$ helm upgrade --install juicefs-csi-driver juicefs/juicefs-csi-driver \
    --namespace kube-system \
    --set kubeletDir=/var/lib/kubelet
	

		$ juicefs format \
    --storage s3 \
    --bucket https://<s3-endpoint>/<bucket> \
    --access-key <access-key> \
    --secret-key <secret-key> \
    redis://:<password>@<redis-host>:6379/1 \
    myjfs
	

		apiVersion: v1
kind: Secret
metadata:
  name: juicefs-sc-secret
  namespace: kube-system
type: Opaque
stringData:
  name: myjfs
  metaurl: redis://:<password>@<redis-host>:6379/1
  storage: s3
  bucket: https://<s3-endpoint>/<bucket>
  access-key: <access-key>
  secret-key: <secret-key>
	

		apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs
provisioner: csi.juicefs.com
volumeBindingMode: Immediate
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
	

		$ kubectl apply -f juicefs-secret.yaml -f juicefs-sc.yaml
	

		apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: juicefs
  resources:
    requests:
      storage: 10Gi
	

		$ kubectl exec writer -- sh -c 'echo "shared across nodes" > /data/hello.txt'
$ kubectl exec reader -- cat /data/hello.txt
shared across nodes
	

Related: Shared storage with NFS for when NFS is the simpler choice · Choose an RWX backend to decide whether you need ReadWriteMany at all.