The problem
In a monolith, "create order + reserve stock + charge card" is one ACID transaction. Split into Order, Inventory and Payment services — each with its own database — and there is no shared transaction. Two-phase commit (2PC) exists but locks resources across services, needs every participant to support it, and blocks when the coordinator fails. Most modern stores (MongoDB, DynamoDB, message brokers) don't participate in 2PC anyway.
The idea
Break the business transaction into local transactions. Each step commits in its own service and triggers the next. If step n fails, run compensating transactions for steps n-1 … 1 to semantically undo them (refund the card, release the stock). The result is eventual consistency with a defined recovery path.
Two ways to coordinate
| Choreography | Orchestration | |
|---|---|---|
| How | Each service reacts to events and emits the next event | A central orchestrator sends commands and tracks state |
| Coupling | Low, but the flow is implicit | Services know the orchestrator only |
| Visibility | Hard — logic is spread across services | Easy — one state machine to inspect |
| Best for | 2–4 simple steps | Long or branching flows, timeouts, retries |
Design rules that matter in production
- Every step must be idempotent. Messages get redelivered;
ReserveStock(orderId)twice must reserve once. - Compensations must be idempotent and must not fail permanently. Retry them until they succeed; alert a human if they can't.
- Publish events reliably — use the Transactional Outbox so "commit DB" and "send message" can't diverge.
- Model semantic locks. Mark records
PENDINGso other requests know the data is mid-saga (lack of isolation is the saga's biggest weakness). - Order steps by risk. Put steps that are hard to compensate (sending an email, calling a third party) last — these are "pivot" and "retriable" transactions.
- Persist saga state so an orchestrator crash resumes rather than restarts.
- A business flow spans services that own their own data
- Eventual consistency is acceptable to the business
- Each step has a meaningful undo (refund, release, cancel)
- You need strict isolation / read-your-writes across the whole flow
- Steps can't be compensated (irreversible side effects early in the flow)
- Everything lives in one database — just use a local transaction
Trade-offs
- ✅ No distributed locks; each service stays autonomous and available.
- ✅ Clear failure semantics — every failure has a defined recovery path.
- ❌ No isolation (the "I" in ACID) — other transactions can see intermediate states. Needs countermeasures: semantic locks, commutative updates, re-reading values.
- ❌ More code: every step needs a compensation, and every handler needs idempotency.
- ❌ Debugging a choreographed saga without tracing is painful — invest in correlation IDs and distributed tracing.
Bulk operations in a multi-tenant CMS behave like a saga: a release deploy touches many entries and locales, and partial failure must be explicit. We modelled per-item outcomes (including a SKIPPED status) rather than all-or-nothing, so the job is resumable and the user sees exactly which items didn't go through and why.
In one line
A saga replaces one distributed transaction with a chain of local transactions plus compensations. I prefer orchestration for anything beyond three steps because the state machine is visible and testable, and I make every step and compensation idempotent with an outbox for reliable events.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Related topics
Transactional Outbox
Write the business change and the outgoing event in the same local transaction, then relay the event to the broker — no more "saved to DB but message lost".
Event-Driven Architecture
Services announce facts ("EntryPublished") and others react asynchronously — decoupling producers from consumers in time, space and knowledge.
Idempotency
Doing an operation twice has the same effect as doing it once. The foundation of safe retries, at-least-once messaging and reliable APIs.
CQRS (Command Query Responsibility Segregation)
Use one model to change data (commands) and a different, optimised model to read it (queries) — often kept in sync by events.