A shared filesystem is the fallback for an app that cannot use object storage, so confirm you need one first with [Choose an RWX backend](/docs/hetzner/apalla/storage/shared/choose-an-rwx-backend). JuiceFS gives you a ReadWriteMany filesystem that pods on different nodes mount read-write at the same time. It stores file data in object storage and file metadata in a database, so the data lives independently of your nodes. That fits Syself Autopilot well: when a node is replaced, there is no volume tied to it to recover. Reach for JuiceFS when an application needs a shared filesystem and cannot use the S3 API directly. If the app can speak S3, [object storage](/docs/hetzner/apalla/storage/object/use-s3-from-workloads) itself is the simpler choice; JuiceFS puts a filesystem in front of that same object storage for apps that need one. JuiceFS is yours to run. The node images ship with FUSE support, which JuiceFS needs to mount the filesystem, so there is nothing to enable on the nodes. You provide the object store, the metadata engine, and the CSI driver. If you would rather not operate the object store and metadata engine yourself, the Syself Consulting team can set JuiceFS up with you or run it as a full service. ## What you need first - **Object storage** for the file data. Run [SeaweedFS](/docs/hetzner/apalla/storage/object/run-seaweedfs) in the cluster, or use any S3-compatible bucket. You need the endpoint, a bucket, and an access key and secret. - **A metadata engine.** JuiceFS keeps the directory tree and file attributes in a fast database. Redis or PostgreSQL both work. This engine is on the write path for every file operation, so run it with the availability your workload needs. If it is down, the filesystem stalls. Install the CSI driver Deploy the JuiceFS CSI driver with Helm: ```console $ helm repo add juicefs https://juicedata.github.io/charts/ $ helm repo update $ helm upgrade --install juicefs-csi-driver juicefs/juicefs-csi-driver \ --namespace kube-system \ --set node.ifPollingKubelet=false \ --set immutable=true ``` > [!WARNING] > If you installed the CSI driver before 1.36, re-run the command above when you upgrade. `node.ifPollingKubelet` and `immutable` are both new requirements: pods can no longer reach the kubelet on port 10250, and the node filesystem is read-only. Without them the driver crashloops and the mount pods never start. Format the filesystem once Formatting writes the filesystem's settings into the metadata engine. Do it once, from anywhere the `juicefs` CLI can reach both the object store and the metadata engine: ```console $ juicefs format \ --storage s3 \ --bucket https:/// \ --access-key \ --secret-key \ redis://:@:6379/1 \ myjfs ``` The last two arguments are the metadata URL (here a Redis database) and the filesystem name (`myjfs`). For PostgreSQL, pass a `postgres://` URL instead. Wire it into Kubernetes Store the same connection details in a Secret the driver reads: ```yaml title="juicefs-secret.yaml" apiVersion: v1 kind: Secret metadata: name: juicefs-sc-secret namespace: kube-system type: Opaque stringData: name: myjfs metaurl: redis://:@:6379/1 storage: s3 bucket: https:/// access-key: secret-key: ``` Create a StorageClass that points at it. Object-backed volumes bind immediately, because there is no node or location to wait for: ```yaml title="juicefs-sc.yaml" apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: juicefs provisioner: csi.juicefs.com volumeBindingMode: Immediate reclaimPolicy: Retain allowVolumeExpansion: true parameters: csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret csi.storage.k8s.io/provisioner-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret csi.storage.k8s.io/node-publish-secret-namespace: kube-system ``` ```console $ kubectl apply -f juicefs-secret.yaml -f juicefs-sc.yaml ``` Create an RWX claim and prove it works Request `ReadWriteMany` on the `juicefs` class: ```yaml title="shared-pvc.yaml" apiVersion: v1 kind: PersistentVolumeClaim metadata: name: shared-data spec: accessModes: - ReadWriteMany storageClassName: juicefs resources: requests: storage: 10Gi ``` Schedule a writer and a reader on different nodes, both mounting the same claim. The anti-affinity rule is what keeps them on separate nodes: ```yaml title="writer-reader.yaml" apiVersion: v1 kind: Pod metadata: name: writer labels: app: juicefs-check spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: {app: juicefs-check} topologyKey: kubernetes.io/hostname containers: - name: app image: nginx volumeMounts: - mountPath: /data name: shared volumes: - name: shared persistentVolumeClaim: claimName: shared-data --- apiVersion: v1 kind: Pod metadata: name: reader labels: app: juicefs-check spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: {app: juicefs-check} topologyKey: kubernetes.io/hostname containers: - name: app image: nginx volumeMounts: - mountPath: /data name: shared volumes: - name: shared persistentVolumeClaim: claimName: shared-data ``` ```console $ kubectl apply -f writer-reader.yaml ``` Once both pods are `Running`, confirm the reader sees the writer's file: ```console $ kubectl exec writer -- sh -c 'echo "shared across nodes" > /data/hello.txt' $ kubectl exec reader -- cat /data/hello.txt shared across nodes ``` > [!NOTE] > The driver runs one mount pod per PVC per node. Several pods on the same node that use the same claim share a single mount; a pod on another node gets its own. Expect a mount pod to appear in `kube-system` for each node that touches the volume. **Related:** [Shared storage with NFS](/docs/hetzner/apalla/storage/shared/nfs-option) for when NFS is the simpler choice ยท [Choose an RWX backend](/docs/hetzner/apalla/storage/shared/choose-an-rwx-backend) to decide whether you need ReadWriteMany at all.