The problem
A downstream service gets slow rather than down. Callers wait, threads and connections pile up, and the caller runs out of resources too. The failure cascades upstream.
The four tools
| Tool | What it does | Key setting |
|---|---|---|
| Timeout | Never wait forever | Set from the dependency's p99, not a guess |
| Retry | Handle transient failures | Exponential backoff + jitter, capped attempts, only for idempotent ops |
| Circuit breaker | Stop calling a failing dependency; probe for recovery | Failure-rate threshold, cool-down window |
| Bulkhead | Separate pools per dependency or tenant so one can't exhaust all | Pool/queue size per dependency |
Plus fallbacks (cached or default response) and load shedding (reject early with 429/503 when saturated).
Gotchas
- Retry storms: retries at every layer multiply load (3 layers × 3 retries = 27 calls). Retry at one layer only, and use retry budgets.
- Retrying non-idempotent writes creates duplicates. See Idempotency.
- A circuit breaker without metrics is a black box. Emit a state-change event every time it opens or closes.
In multi-tenant platforms, bulkhead by tenant for heavy jobs (bulk publish, imports). One enterprise tenant's 200k-item job shouldn't starve everyone else's queue.
Cheatsheet
The whole topic on one page. Click to open full screen.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Related topics
Idempotency
Doing an operation twice has the same effect as doing it once. The foundation of safe retries, at-least-once messaging and reliable APIs.
Microservices Architecture
Structure a system as independently deployable services, each owning a business capability and its data, communicating over the network.
Observability — Logs, Metrics, Traces
Know what your system is doing in production — the three pillars, OpenTelemetry, SLIs/SLOs, and alerting on symptoms not causes.