Nodes communicate with each other over the public internet, so the network between two nodes occasionally drops packets. See [MTU and the VXLAN tunnel](/docs/hetzner/apalla/network/egress/mtu-and-tunnel-behavior). The cause is usually a degraded switch or router, not an outage, and the symptom is requests that fail for no clear reason, often "connection reset by peer", and recover without intervention. Without any handling, one dropped packet is one error your user sees.

Istio is a service mesh: it puts a proxy (a sidecar) next to each pod and routes the pod's traffic through it. That allows it to retry the one request that failed and to stop sending traffic to a pod that keeps failing (outlier detection), without any change to your application, so the user gets a slightly slower response instead of an error. This page shows how to set up both, hop by hop. To decide whether you need a mesh at all, and how Istio compares to other options, see [Choose a service mesh](/docs/hetzner/apalla/network/mesh/choose-a-service-mesh).

> [!NOTE]
> Istio is not shipped or managed by the platform. Install it yourself, and use **sidecar mode**. Ambient mode is not supported, and the settings here are written for sidecar mode. See [Use sidecar mode, not ambient](/docs/hetzner/apalla/network/mesh/choose-a-service-mesh#use-sidecar-mode-not-ambient) for why.

## Per-hop resilience on the caller side

A request crosses several nodes on its way through your application, and every hop travels through network hardware where a packet can drop. Resilience is configured on the caller side of each hop, and it protects only that one hop. You therefore need a policy at every service-to-service hop, not just at the ingress.

```mermaid
flowchart LR
  I["Ingress gateway"]:::app -->|retry| A["Service A"]:::app
  A -->|retry| B["Service B"]:::app
  B -->|driver retry, not Istio| D[(Database)]:::data
```

| Hop                              | In the mesh?                     | What protects it                                               |
| -------------------------------- | -------------------------------- | -------------------------------------------------------------- |
| Ingress gateway to first service | Yes                              | Istio retries and outlier detection                            |
| Service to service               | Yes                              | Istio retries and outlier detection                            |
| Application to database          | No (no sidecar on database pods) | Application code: driver retries, pool health checks, timeouts |

Only retry when the request never reached the application, for example a connection reset before the request was delivered. Never retry an error response from the application, because the server may already have done the work.

## Configure retries per hop

A retry saves the request that fails right now: Istio resends it, and it can land on a different pod and take a different path through the network. Add a `VirtualService` on the caller side of each service-to-service hop:

```yaml title="virtualservice.yaml"
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts: [backend]
  http:
    - route:
        - destination: {host: backend}
      timeout: 3s
      retries:
        attempts: 3
        perTryTimeout: 800ms
        retryOn: connect-failure,refused-stream,reset-before-request
```

- **`retryOn`** lists only causes where the request never reached the application. Use `reset-before-request`, not plain `reset`: `reset` also retries after the request was delivered, which can repeat a write such as a payment or an order.
- **`perTryTimeout` must be smaller than `timeout`.** If they are equal, there is no time left for a second attempt and retries silently never fire.
- **Keep `attempts` low.** Every retry multiplies load on the downstream service, so too many retries can make a struggling service worse.

## Configure outlier detection per service

When one pod keeps returning errors, outlier detection stops sending it traffic for a while and shifts it to the healthy pods. Add a `DestinationRule` per service:

```yaml title="destinationrule.yaml"
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: backend
spec:
  host: backend
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
```

- **`consecutive5xxErrors`** ejects a pod after five 5xx responses in a row. **`interval`** is how often Istio runs that check, every 10 seconds here.
- **`baseEjectionTime`** is how long a pod stays out before Istio tries it again, growing each time it is re-ejected.
- **`maxEjectionPercent: 50`** stops Istio from removing more than half the pool at once, so the service keeps a quorum serving even when several pods struggle.

```mermaid
flowchart LR
  C["Caller"]:::app --> P1["Pod 1 healthy"]:::app
  C --> P2["Pod 2 healthy"]:::app
  C -. ejected after 5 errors .-> P3["Pod 3 failing"]:::app
```

Both settings need at least two replicas. With a single replica there is no healthy pod to retry against or shift traffic to.

## Keep the database hop out of the mesh

Do not attach a sidecar to database pods. Database connections are long-lived and are not plain HTTP, so an Istio sidecar can break them with idle-timeout resets or protocol confusion. Mark database pods so Istio skips them:

```yaml
metadata:
  labels:
    sidecar.istio.io/inject: "false"
```

Because the mesh is not on this hop, resilience here belongs in the application:

- **Driver retries for idempotent reads only.** Do not auto-retry writes or anything mid-transaction; they may not be safe to repeat.
- **Pool health checks** that test a connection before use, so a stale connection is discarded rather than handed to a query.
- **Explicit connect and statement timeouts**, so a stalled connection fails fast instead of hanging.

TLS from the application to the database is the database driver's responsibility, not Istio's.

## Checklist when a service still fails

- Every hop has its own retry policy, not just the ingress.
- `perTryTimeout` is smaller than the overall `timeout`, or retries never fire.
- Write routes only retry on "never reached the app" causes, so a charge or an order is never repeated.
- The service has more than one replica, or there is nowhere to shift traffic.
- Retry `attempts` are kept low; retries add load and can make a struggling service worse.
- The database hop is handled in the application, not Istio.

If packet drops are constant rather than occasional, retries are masking a genuine network fault. Stop tuning retries and identify the root cause: start with [Debug node networking](/docs/hetzner/apalla/network/debug/debug-node-networking).

## Where to go next

- [Choose a service mesh](/docs/hetzner/apalla/network/mesh/choose-a-service-mesh)
- [See flows with Hubble](/docs/hetzner/apalla/observability/network-flows/see-flows-with-hubble)
- [Debug node networking](/docs/hetzner/apalla/network/debug/debug-node-networking)
- [Survive node drains](/docs/hetzner/apalla/workloads/production/survive-node-drains)