Longhorn stores the replicas of each volume on the disks of your nodes. Syself Autopilot replaces nodes during updates, upgrades and self-healing, and so the Longhorn data on a replaced node is lost. This page shows how to replace those nodes yourself and move their data to other nodes first, so your volumes stay healthy.

This page follows the Longhorn article [Graceful Longhorn Node Eviction Before Cluster API (CAPI) Node Replacement](https://longhorn.io/kb/how-to-evict-node-during-capi-node-rolling-replacement/). It assumes Longhorn is already installed. To install it, see the [Longhorn documentation](https://longhorn.io/docs/latest/deploy/install/).

## Keep Longhorn data when a node is replaced

Usually, the Longhorn data on a node is lost when an update or upgrade replaces the node. To prevent this, switch the pool that stores Longhorn data to manual replacement. Then you decide when each node is replaced, and you move its data to other nodes first. Longhorn calls this an eviction. The sections below show both steps.

Without these steps, the data is lost for two reasons. First, Syself Autopilot drains a node for at most 180 seconds and then deletes it, even if Longhorn has not moved its data yet. This drain and delete comes from Cluster API, so it is not specific to Syself Autopilot. Second, the data does not survive on the node. Longhorn stores it in `/var/lib/longhorn` by default, which is on the OS disk. Syself Autopilot rewrites that disk every time it provisions a node.

Self-healing works differently. When a node fails, it is replaced automatically. This also happens in a pool with manual replacement. There is no time to evict the node first. Longhorn then rebuilds the lost replicas from the copies on other nodes, so keep at least two replicas per volume. One exception: during an update, a failed node that has not been updated yet is not replaced automatically. Delete its machine yourself.

On bare metal, self-healing reprovisions a node that does not recover from a reboot, which wipes it. To prevent that, turn off the health checks for the Longhorn pool. See [Turn off health checks for a pool](/docs/hetzner/apalla/servers-and-nodes/maintenance/machine-health-checks-and-remediation#turn-off-health-checks-for-a-pool).

## Before you start

Before evicting a node, make sure the other nodes have enough free disk space to store its Longhorn replicas and backing images.

By default, Longhorn keeps each replica of a volume on a different node. So you need at least as many other nodes as your volumes have replicas, or the eviction cannot finish. See [Scheduling](https://longhorn.io/docs/latest/nodes-and-volumes/nodes/scheduling/) in the Longhorn documentation.

## Switch your Longhorn pool to manual replacement

Keep all nodes that store Longhorn data in one pool. The rollout strategy is set per pool, so one setting then decides how all of them are replaced.

For that pool, set the rollout strategy to `OnDelete` in your `Cluster` object:

```yaml
spec:
  topology:
    workers:
      machineDeployments:
        - name: <pool-name>
          rollout:
            strategy:
              type: OnDelete
```

With `OnDelete`, an update or upgrade does not replace the nodes of that pool. Each node keeps running until you delete its machine.

## Replace a node

Replace the nodes of a Longhorn pool one at a time. Run the Longhorn commands against the workload cluster.

<Steps>

<Step>Find the Longhorn node</Step>

Longhorn names its nodes after the Kubernetes nodes:

```console
$ kubectl get nodes.longhorn.io -n longhorn-system
```

<Step>Request the eviction</Step>

Stop Longhorn from placing new replicas on the node and ask it to move the existing ones:

```console
$ kubectl patch node.longhorn.io <node-name> \
    -n longhorn-system --type=merge \
    -p '{"spec":{"allowScheduling":false,"evictionRequested":true}}'
```

<Step>Wait until the node is empty</Step>

Check how many replicas and backing images are still on each disk of the node:

```console
$ kubectl get node.longhorn.io <node-name> -n longhorn-system -o json \
    | jq '.status.diskStatus | to_entries[] | {disk: .key, scheduledReplicas: (.value.scheduledReplica | length), scheduledBackingImages: (.value.scheduledBackingImage | length)}'
```

The eviction is complete when every disk shows `scheduledReplicas: 0` and `scheduledBackingImages: 0`. Longhorn then no longer needs the node, and you can delete its machine.

Depending on the data size and network speed, this takes minutes to hours.

<Step>Delete the machine</Step>

On the management cluster, find the machine of the node and delete it:

```console
$ kubectl get machines -n <namespace>
$ kubectl delete machine <machine-name> -n <namespace>
```

Syself Autopilot creates a new machine, and a new node joins the cluster. A bare metal node keeps its name if it comes back on the same server.

</Steps>

## Automating eviction is not supported

The Longhorn article also describes an automated method. A custom controller in the management cluster adds a pre-termination hook to each machine that is being deleted. The hook holds the deletion until the eviction is complete.

Syself Autopilot does not support this method. Use the manual steps above.

Related: [cluster upgrades](/docs/hetzner/apalla/concepts/operations/cluster-upgrades) for how a rollout replaces nodes.