Handle unreliable networks with Istio

Nodes communicate with each other over the public internet, so the network between two nodes occasionally drops packets. See MTU and the VXLAN tunnel . The cause is usually a degraded switch or router, not an outage, and the symptom is requests that fail for no clear reason, often "connection reset by peer", and recover without intervention. Without any handling, one dropped packet is one error your user sees.

Istio is a service mesh: it puts a proxy (a sidecar) next to each pod and routes the pod's traffic through it. That allows it to retry the one request that failed and to stop sending traffic to a pod that keeps failing (outlier detection), without any change to your application, so the user gets a slightly slower response instead of an error. This page shows how to set up both, hop by hop. To decide whether you need a mesh at all, and how Istio compares to other options, see Choose a service mesh .

Per-hop resilience on the caller side 

A request crosses several nodes on its way through your application, and every hop travels through network hardware where a packet can drop. Resilience is configured on the caller side of each hop, and it protects only that one hop. You therefore need a policy at every service-to-service hop, not just at the ingress.

flowchart LR
  I["Ingress gateway"]:::app -->|retry| A["Service A"]:::app
  A -->|retry| B["Service B"]:::app
  B -->|driver retry, not Istio| D[(Database)]:::data

Only retry when the request never reached the application, for example a connection reset before the request was delivered. Never retry an error response from the application, because the server may already have done the work.

Configure retries per hop 

A retry saves the request that fails right now: Istio resends it, and it can land on a different pod and take a different path through the network. Add a VirtualService on the caller side of each service-to-service hop:

				apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts: [backend]
  http:
    - route:
        - destination: {host: backend}
      timeout: 3s
      retries:
        attempts: 3
        perTryTimeout: 800ms
        retryOn: connect-failure,refused-stream,reset-before-request
			

• retryOn lists only causes where the request never reached the application. Use reset-before-request, not plain reset: reset also retries after the request was delivered, which can repeat a write such as a payment or an order.
• perTryTimeout must be smaller than timeout. If they are equal, there is no time left for a second attempt and retries silently never fire.
• Keep attempts low. Every retry multiplies load on the downstream service, so too many retries can make a struggling service worse.

Configure outlier detection per service 

When one pod keeps returning errors, outlier detection stops sending it traffic for a while and shifts it to the healthy pods. Add a DestinationRule per service:

				apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: backend
spec:
  host: backend
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
			

• consecutive5xxErrors ejects a pod after five 5xx responses in a row. interval is how often Istio runs that check, every 10 seconds here.
• baseEjectionTime is how long a pod stays out before Istio tries it again, growing each time it is re-ejected.
• maxEjectionPercent: 50 stops Istio from removing more than half the pool at once, so the service keeps a quorum serving even when several pods struggle.

flowchart LR
  C["Caller"]:::app --> P1["Pod 1 healthy"]:::app
  C --> P2["Pod 2 healthy"]:::app
  C -. ejected after 5 errors .-> P3["Pod 3 failing"]:::app

Both settings need at least two replicas. With a single replica there is no healthy pod to retry against or shift traffic to.

Keep the database hop out of the mesh 

Do not attach a sidecar to database pods. Database connections are long-lived and are not plain HTTP, so an Istio sidecar can break them with idle-timeout resets or protocol confusion. Mark database pods so Istio skips them:

				metadata:
  labels:
    sidecar.istio.io/inject: "false"
			

Because the mesh is not on this hop, resilience here belongs in the application:

• Driver retries for idempotent reads only. Do not auto-retry writes or anything mid-transaction; they may not be safe to repeat.
• Pool health checks that test a connection before use, so a stale connection is discarded rather than handed to a query.
• Explicit connect and statement timeouts, so a stalled connection fails fast instead of hanging.

TLS from the application to the database is the database driver's responsibility, not Istio's.

Checklist when a service still fails 

• Every hop has its own retry policy, not just the ingress.
• perTryTimeout is smaller than the overall timeout, or retries never fire.
• Write routes only retry on "never reached the app" causes, so a charge or an order is never repeated.
• The service has more than one replica, or there is nowhere to shift traffic.
• Retry attempts are kept low; retries add load and can make a struggling service worse.
• The database hop is handled in the application, not Istio.

If packet drops are constant rather than occasional, retries are masking a genuine network fault. Stop tuning retries and identify the root cause: start with Debug node networking .

Where to go next 

• Choose a service mesh
• See flows with Hubble
• Debug node networking
• Survive node drains