Saga and outbox: a transaction without a distributed transaction

Inside one database, atomicity is free: BEGIN, a handful of statements, COMMIT. Either everything happens or nothing does.

The moment the operation crosses two databases, or two services, that guarantee disappears. And what most teams do is pretend it is still there, which produces the hardest category of bug to reproduce that exists.

Why 2PC is not the answer#

The academic solution is two phase commit. A coordinator asks every participant "can you do this?"; if they all say yes, it tells them to commit.

It works. And it has a problem that makes it impractical: if the coordinator dies between the two phases, the participants sit there with locks open, waiting indefinitely, not knowing whether to commit or roll back.

It is blocking, it requires every participant to support the protocol, and it creates a single point of failure in exactly the component whose job is to guarantee reliability.

It exists, it works in controlled contexts, and almost nobody wants it.

Saga: break it into compensable steps#

The practical answer is the saga. You break the operation into a sequence of local transactions, each with a matching compensation.

• Debited account A and the credit to account B failed? Run the reversal on A.
• Reserved the stock and the payment failed? Release the reservation.
• Created the order and the invoice failed? Cancel the order.

What a saga gives you: no distributed locks. Every step is an ordinary local transaction, in the database that already owns that data.

What it takes away: real atomicity. There is a window where the system sits in an observable intermediate state. Someone can query and see an order with no invoice.

And here is the part that usually stalls the discussion: your product has to tolerate that. It usually does, because the real world works that way anyway: the seat is reserved before the payment clears, the order is created before the invoice is issued.

When it does not tolerate it, the answer is not 2PC. It is to reconsider the service boundary: if two things genuinely have to be atomic, maybe they belong in the same database.

Two styles of saga#

Choreography: each service reacts to events and publishes its own. Nobody coordinates. It is simple to start with and becomes hard to follow past three or four steps: nobody can answer "what is the full flow?" by looking in one place.

Orchestration: there is an orchestrator that knows the flow and calls each step. Easier to understand, to monitor and to debug. It introduces a central component, which has to be resilient and keep persisted state.

The practical recommendation: choreography for flows of two or three steps; orchestration from there on. And if you pick orchestration, persist the saga state, because an in-memory orchestrator that loses state on restart leaves sagas hanging half done.

The outbox pattern#

Now the most common problem of all, and the one that causes the most silent bugs:

saveOrder(order)
publishEvent("OrderCreated", order)

There is no transaction covering both. Two bad scenarios:

• It saved and the publish failed: the order exists and nobody was notified.
• It published and the commit failed: half the system reacted to an order that does not exist.

The real life symptom is that "every now and then an order does not show up in the other service and we could not reproduce it".

The outbox pattern solves it with an idea that is almost embarrassingly simple:

In the same transaction where you write the order, also write a row into an outbox table.

BEGIN
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox (type, payload, created_at) VALUES (...);
COMMIT

One local transaction, atomic, no coordinator, no special protocol.

Then a separate process reads the outbox table, publishes to the queue and marks the row as published. If it dies halfway, it picks up where it stopped. If it publishes twice (and eventually it will), the consumer has to be idempotent.

The guarantee stops being "we hope it works" and becomes demonstrable: if it is in the database, it is in the outbox; if it is in the outbox, it will be published.

The CDC variant#

There is a version that skips even the reader process: change data capture. Tools like Debezium read the database's own replication log and publish the changes.

The outbox table becomes just one more captured table, and you do not even write the publisher.

The cost: one more piece of infrastructure to operate, and a specific caution on Postgres: the replication slot. If the consumer stops, the database holds the WAL for it, and the disk fills up. Monitor slot lag; it has taken down plenty of production databases.

Idempotency: the mandatory piece#

None of this works without idempotent consumers.

The standard technique: every event carries a unique identifier. The consumer keeps a table of already processed events and ignores repeats. Or, better still, the operation itself is naturally idempotent: an UPSERT instead of an INSERT, an assignment instead of an increment.

What to do tomorrow#

If you have an operation that writes to the database and publishes an event, you have this bug. It just has not annoyed you enough yet.

The shortest path: create the outbox table, move the publish inside the transaction, write the publisher in fifty lines.