Add retries and outlier detection

Retries and outlier detection are the Istio settings that absorb an unreliable network. A retry saves the request that fails immediately by sending it again. Outlier detection ejects a pod that keeps failing, so callers stop reaching it and traffic moves to the healthy pods. Use both: they solve different problems. This page assumes Istio is installed in sidecar mode; see Handle unreliable networks with Istio for the model, and Choose a service mesh for whether you need a mesh in the first place.

Configure retries per hop#

When a request fails on a hop, Istio resends it. The retry can land on a different pod and take a different path through the network, so the user gets a slightly slower response instead of an error. Add a VirtualService on the caller side of each service-to-service hop:

		apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts: [backend]
  http:
    - route:
        - destination: {host: backend}
      timeout: 3s
      retries:
        attempts: 3
        perTryTimeout: 800ms
        retryOn: connect-failure,refused-stream,reset-before-request
	

• retryOn: connect-failure,refused-stream,reset-before-request means "the request never reached the application", the only cases that are safe to repeat. Use reset-before-request, not plain reset: reset also fires when the connection drops after the request was delivered, so the server may have already acted on it, while reset-before-request only retries a reset that happened before the request was sent. On a route that writes (charging a card, creating an order), keep it to exactly these causes so a partial request is never replayed.
• perTryTimeout must be smaller than timeout. If they are equal, there is no time left for a second attempt and retries silently never fire.
• Keep attempts low. Every retry multiplies load on the downstream service, so too many retries can make a struggling service worse.

Configure outlier detection per service#

Outlier detection watches real request outcomes. When one pod keeps returning errors, Istio stops sending it traffic for a while, then gradually tries again. Traffic shifts to the healthy pods on its own. Add a DestinationRule per service:

		apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: backend
spec:
  host: backend
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
	

• consecutive5xxErrors and interval decide how fast a bad pod is ejected: five 5xx responses inside a 10-second window here.
• baseEjectionTime is how long a pod stays out before Istio tries it again, growing each time it is re-ejected.
• maxEjectionPercent: 50 stops Istio from removing more than half the pool at once, so the service keeps a quorum serving even when several pods struggle.

flowchart LR
  C["Caller"] --> P1["Pod 1 healthy"]
  C --> P2["Pod 2 healthy"]
  C -. ejected after 5 errors .-> P3["Pod 3 failing"]

The caller keeps reaching the healthy pods and stops sending to the failing one until baseEjectionTime passes and Istio tries it again.

Both settings need more than one replica. With a single replica there is no healthy pod to retry against or shift traffic to, so run at least two. Together, retries save the request that fails immediately, and outlier detection steers traffic away from a pod that is consistently unhealthy.

Where to go next#

• Run a production-ready workload