Local storage keeps data on the node, so it is faster than network storage, and on [bare-metal machines](/docs/hetzner/apalla/concepts/internals/bare-metal-and-cloud) it is cheap too. Syself Autopilot handles the lifecycle: data persists through cluster updates and machine re-provisioning, which suits storage-intensive workloads such as databases. This one-time setup configures your cluster and machines to use local storage with TopoLVM.

> [!WARNING]
> Most of the procedure described in this guide will be automated in the upcoming release of Syself Autopilot. Please stick to the configuration options described here to guarantee compatibility.

{#if release.name === 'hetzner-apalla-1-29-v1'}

## Prerequisites

We need to add an annotation to your Cluster object to keep the hostnames of your bare-metal nodes constant. This is necessary so that Kubernetes can keep track of your volumes and doesn't lose them after upgrades or when your machines are re-provisioned.

Follow the steps for your scenario:

<Tabs items={['New cluster', 'Cluster w/o bare-metal', 'Cluster w/ bare-metal']}>

<TabItem value="New cluster">

You should follow the steps described here if you want to setup local storage in a new cluster.

1. Add a label `capi.syself.com/constant-bare-metal-hostname='true'` in your Cluster object. If you are unsure how to create a new cluster, refer to [Creating Clusters](/docs/hetzner/apalla/getting-started/create-your-first-cluster).

   The `metadata` section of your Cluster should look something like this:

   ```yaml title="cluster.yaml"
   metadata:
     annotations:
       capi.syself.com/constant-bare-metal-hostname: "true"
   ```

1. Add your bare metal machines. If you are unsure how to do it, follow the steps in [Adding Bare Metal Servers](/docs/hetzner/apalla/servers-and-nodes/provision/add-bare-metal-servers).

   > [!NOTE]
   > For using local storage with the setup described here, the machines should not use RAID.

</TabItem>

<TabItem value="Cluster w/o bare-metal">

You should follow the steps described here if you want to setup local storage in an already-existing cluster that has no bare-metal workers.

1. Annotate your Cluster object

   ```console
   $ kubectl annotate cluster mycluster capi.syself.com/constant-bare-metal-hostname='true'
   ```

1. Add your bare metal machines. If you are unsure how to do it, follow the steps in [Adding Bare Metal Servers](/docs/hetzner/apalla/servers-and-nodes/provision/add-bare-metal-servers).

   > [!NOTE]
   > For using local storage with the setup described here, the machines should not use RAID.

</TabItem>

<TabItem value="Cluster w/ bare-metal">

You should follow the steps described here if you want to setup local storage in an already-existing cluster that has bare-metal machines deployed to it.

> [!NOTE]
> For using local storage with the setup described here, your bare-metal machines should not use RAID.

1. Annotate your Cluster object:

   ```console
   $ kubectl annotate cluster mycluster capi.syself.com/constant-bare-metal-hostname='true'
   ```

1. Roll-out your bare-metal machines:

   ```console
   $ clusterctl alpha rollout restart machinedeployment/mycluster-md-bm
   ```

1. Check if your machines were correctly reprovisioned:

   ```console
   $ kubectl get machines
   ```

   The machines in the output should have their Server IDs in their node names. If you are unsure what is the Server ID of your machine, you can check that in the Hetzner Robot / Servers tab or in the HetznerBareMetalHosts resource.

<Callout type="note" title="Installing clusterctl">

If you don't have `clusterctl` installed, get it with one of the commands below:

<Tabs id="operating-systems" items={['Linux', 'macOS', 'Windows']}>

<TabItem value="Linux">

```console title="x86-64"
$ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-linux-amd64 -o clusterctl
$ sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl
```

```console title="ARM64"
$ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-linux-arm64 -o clusterctl
$ sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl
```

</TabItem>

<TabItem value="macOS">

```console title="Intel"
$ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-darwin-amd64 -o clusterctl
$ chmod +x ./clusterctl
$ sudo mv ./clusterctl /usr/local/bin/clusterctl
```

```console title="Apple Silicon"
$ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-darwin-arm64 -o clusterctl
$ chmod +x ./clusterctl
$ sudo mv ./clusterctl /usr/local/bin/clusterctl
```

</TabItem>

<TabItem value="Windows">

```console title="Windows"
$ curl.exe -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.2/clusterctl-windows-amd64.exe -o clusterctl.exe
```

</TabItem>

</Tabs>

</Callout>

</TabItem>

</Tabs>

{/if}

## 1. Deploy cert-manager

You need to have `cert-manager` version `v1.7.0` or higher installed on your cluster as a dependency of TopoLVM. Install it with the following command:

```console
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm template --namespace=kube-system cert-manager jetstack/cert-manager --set installCRDs=true | kubectl apply -n kube-system -f -
```

You can follow the [official guide](https://helm.sh/docs/intro/install/) on installing `helm` if you don't have it.

> [!WARNING]
> To guarantee compatibility with future Syself Autopilot releases, **don't** use `helm install`, as our automation will create the resources directly instead of installing charts.

## 2. Deploy TopoLVM

We use the TopoLVM CSI driver for local storage on bare-metal. You can follow the steps below to deploy it to your workload cluster.

1. Add the Syself helm repository:

   ```console
   $ helm repo add syself https://charts.syself.com
   $ helm repo update
   ```

2. Template the TopoLVM chart and apply it to the cluster:

   > [!WARNING]
   > Do **not** use `helm install`. This chart will be added as a base feature in later versions of Syself Autopilot, so stick to the installation steps shown here to guarantee compatibility. Use the command below to install it with `helm template` and `kubectl apply` to the `kube-system` namespace.

   ```console
   $ helm template --namespace=kube-system csi-local syself/topolvm | kubectl apply -n kube-system -f -
   ```

   Now the storage space in your bare-metal server is exposed to your cluster via the `local-nvme` Storage Class:

   ```console
   $ kubectl get storageclasses
   NAME                  PROVISIONER         RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
   local-hdd             topolvm.io          Retain          WaitForFirstConsumer   true
   local-nvme            topolvm.io          Retain          WaitForFirstConsumer   true
   local-ssd             topolvm.io          Retain          WaitForFirstConsumer   true
   standard (default)    csi.hetzner.cloud   Retain          WaitForFirstConsumer   true
   ```

## 3. Configure your servers

In this step, you'll define the _physical volumes_ and _volume groups_ in your disks to be used by TopoLVM.

> [!NOTE]
> >
> We support all three types of disks: HDD, SATA SSD, and NVMe SSD. If your server, for example, only has NVMe, you should only follow the steps for NVMe and cannot use the storage classes `local-ssd` or `local-hdd`.
>
> If your server has NVMe, SSD, and HDD, you can use all three storage classes.

1. Access your server via ssh:

   ```console
   $ ssh -i path-to-your/ssh-key -p 100 root@<machine-ip>
   ```

1. List your disks with `lsblk`:

   ```console
   $ lsblk
   NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   nvme1n1     259:0    0 476.9G  0 disk
   nvme0n1     259:1    0 476.9G  0 disk
   |-nvme0n1p1 259:2    0   512M  0 part /boot/efi
   |-nvme0n1p2 259:3    0     1G  0 part /boot
   `-nvme0n1p3 259:4    0 475.4G  0 part /
   ```

1. Identify if the disk(s) you want to use is an HDD, SATA SSD, or NVMe SSD.

   > [!WARNING]
   > Don't use your OS disk (`nvme0n1` in the above output), as this can lead to data loss.

   > [!TIP]
   > To identify the type of disk you have, you can look at the first column `NAME` and third column `RM` of the `lsblk` output.
   >
   > - An NVMe disk will have `nvme` at the beginning of its name, otherwise:
   > - A SATA SSD disk will have the value 0 in the `RM` column.
   > - A HDD disk will have the value 1 in the `RM` column.

1. Create a physical volume (point to every disk in your server where you want to store data) with `pvcreate /dev/[disk-name]`. For example:

   ```console
   $ pvcreate /dev/nvme1n1
   ```

1. Map the disks to the appropriate volume group type with `vgcreate vg-[type] /dev/[disk-name] /dev/[other-disk]`. For example:

   ```console
   $ vgcreate vg-nvme /dev/nvme1n1
   ```

   If you missed a disk and want to add it later, extend the volume group with `vgextend vg-[type] /dev/[new-disk]`. For example:

   ```console
   $ vgextend vg-nvme /dev/nvme2n1
   ```

   <Callout type="note" title="Available volume group types">
   - For NVMe disks you use: `vg-nvme`
   - For SATA SSD disks you use: `vg-ssd`
   - For HDD disks you use: `vg-hdd`

   </Callout>

1. Repeat the previous steps for every disk you want to use.
1. After adding all disks to their respective volume groups, create a thin provisioned logical volume for each volume group. This step is done once per volume group, not per disk: Create a thin provisioned logical volume with `lvcreate --thinpool pool-[type] --extents 100%FREE vg-[type]`.

<Tabs id="storage-types" items={['NVMe', 'SSD', 'HDD']}>

<TabItem value="NVMe">

```console
$ lvcreate --thinpool pool-nvme --extents 100%FREE vg-nvme
```

</TabItem>

<TabItem value="SSD">

```console
$ lvcreate --thinpool pool-ssd --extents 100%FREE vg-ssd
```

</TabItem>

<TabItem value="HDD">

```console
$ lvcreate --thinpool pool-hdd --extents 100%FREE vg-hdd
```

</TabItem>

</Tabs>

## 4. Use it!

You can use the newly created Storage Classes in the same way you would use any other.

<Callout type="note" title="Available storage classes">

By default, you have Storage Classes for all three disk types available in your cluster:

- For NVMe disks: `local-nvme`
- For SATA SSD disks: `local-ssd`
- For HDD disks: `local-hdd`

If you use a Storage Class for a disk type unavailable in your machine volume groups, your workload will be stuck at provisioning. We include all three to make your cluster ready for any new disks you might add in the future.

</Callout>

If you want to test your new setup:

1. Create a `pv-claim.yaml` file with the following content:

   ```yaml title="pv-claim.yaml"
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pv-claim
   spec:
     storageClassName: local-nvme
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
   ```

1. And apply it with `kubectl apply -f pv-claim.yaml`.
1. Create a `pod.yaml` file with the following content:

   ```yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: pv-pod
   spec:
     volumes:
       - name: pv-storage
         persistentVolumeClaim:
           claimName: pv-claim
     containers:
       - name: pv-container
         image: nginx
         ports:
           - containerPort: 80
             name: http-server
         volumeMounts:
           - mountPath: /usr/share/nginx/html
             name: pv-storage
   ```

1. And apply it with `kubectl apply -f pod.yaml`.

   Now, all the data stored in the container under `/usr/share/nginx/html` will be in `/mnt/data` on your machine.

   > [!NOTE]
   > In this example, we used the `local-nvme` class, but you can also use `local-hdd` and `local-ssd` too if your servers have disks of those types attached.

1. Create a test file in your pod:

   ```console
   $ kubectl exec -it pv-pod -- /bin/sh
   $ echo 'Hi from Kubernetes to bare metal!' > /usr/share/nginx/html/hi.txt
   ```

1. Now delete the pod, so we are sure the storage is persistent, and not ephemeral: `kubectl delete pod pv-pod`

1. Apply the pod again with `kubectl apply -f pod.yaml`. Show the content of the `hi.txt` file:

   ```console
   $ cat /mnt/data/hi.txt
   Hi from Kubernetes to bare metal!
   ```