Nodes communicate with each other over the public internet, so the network between two nodes occasionally drops packets. The cause is usually a degraded switch or router, not an outage, and the symptom is requests that fail for no clear reason, often "connection reset by peer", and recover without intervention. Without any handling, one dropped packet is one error your user sees.

Istio is a service mesh: it puts a proxy (a sidecar) next to each pod and routes the pod's traffic through it. That allows it to retry the one request that failed, without any change to your application, so the user gets a slightly slower response instead of an error. Istio is one option; [Choose a service mesh](/docs/hetzner/apalla/network/mesh/choose-a-service-mesh) covers when you need a mesh at all and how Istio compares.

> [!NOTE]
> Istio is not shipped or managed by the platform. Install it yourself, and use **sidecar mode**. Ambient mode is not supported, and the settings here are written for sidecar mode. See [Use sidecar mode, not ambient](/docs/hetzner/apalla/network/mesh/choose-a-service-mesh#use-sidecar-mode-not-ambient) for why.

## Per-hop resilience on the caller side

A request crosses several nodes on its way through your application, and every hop travels through network hardware where a packet can drop. Resilience is configured on the caller side of each hop, and it protects only that one hop. You therefore need a policy at every service-to-service hop, not just at the ingress.

```mermaid
flowchart LR
  I["Ingress gateway"] -->|retry| A["Service A"]
  A -->|retry| B["Service B"]
  B -->|driver retry, not Istio| D["Database"]
```

Each arrow is one hop, and each needs its own resilience on the caller side. The database hop sits outside the mesh, so its retries live in the application, not Istio.

| Hop                              | In the mesh?                     | What protects it                                               |
| -------------------------------- | -------------------------------- | -------------------------------------------------------------- |
| Ingress gateway to first service | Yes                              | Istio retries and outlier detection                            |
| Service to service               | Yes                              | Istio retries and outlier detection                            |
| Application to database          | No (no sidecar on database pods) | Application code: driver retries, pool health checks, timeouts |

Only retry when the request never reached the application, so retrying is safe. A connection reset before the request is delivered is such a case: nothing ran on the other side yet. A reset that arrives after the request was sent is not safe to replay, because the server may already have acted on it. Never retry a normal error response from the application; the server may already have performed the work. The exact retry and ejection settings live in [Add retries and outlier detection](/docs/hetzner/apalla/network/mesh/retries-and-outlier-detection).

## Keep the database hop out of the mesh

Do not attach a sidecar to database pods. Database connections are long-lived and are not plain HTTP, so an Istio sidecar can break them with idle-timeout resets or protocol confusion. Mark database pods so Istio skips them:

```yaml
metadata:
  labels:
    sidecar.istio.io/inject: "false"
```

Because the mesh is not on this hop, resilience here belongs in the application:

- **Driver retries for idempotent reads only.** Do not auto-retry writes or anything mid-transaction; they may not be safe to repeat.
- **Pool health checks** that test a connection before use, so a stale connection is discarded rather than handed to a query.
- **Explicit connect and statement timeouts**, so a stalled connection fails fast instead of hanging.

TLS from the application to the database is the database driver's responsibility, not Istio's.

## Checklist when a service still fails

- Every hop has its own retry policy, not just the ingress.
- `perTryTimeout` is smaller than the overall `timeout`, or retries never fire.
- Write routes only retry on "never reached the app" causes, so a charge or an order is never repeated.
- The service has more than one replica, or there is nowhere to shift traffic.
- Retry `attempts` are kept low; retries add load and can make a struggling service worse.
- The database hop is handled in the application, not Istio.

If packet drops are constant rather than occasional, retries are masking a genuine network fault. Stop tuning retries and identify the root cause: start with [Debug node networking](/docs/hetzner/apalla/network/debug/debug-node-networking).

## Where to go next

- [Add retries and outlier detection](/docs/hetzner/apalla/network/mesh/retries-and-outlier-detection)