CAP, PACELC and the consistency models

"By the CAP theorem, we picked AP."

That sentence shows up in every architecture discussion, and most of the time the person saying it does not know what they are saying. Not out of bad faith, but because the popularised version of the theorem is a simplification that lost its meaning along the way.

The correct statement#

The popular version is "pick two out of consistency, availability and partition tolerance". It suggests you sit down at a table and choose.

But a network partition is not a choice. It is something that happens to you. A cut cable, a faulty switch, an isolated availability zone, a wrong firewall rule.

The correct statement is:

When a network partition exists, you have to choose between continuing to answer at the risk of divergent data, or stopping answering to preserve consistency.

That is all. And note the "when": outside a partition, CAP says absolutely nothing.

A CP system: during the partition, the minority side refuses writes. You lose availability and keep correctness. That is the behaviour of etcd, ZooKeeper, Consul, and of a Postgres with synchronous replication.

An AP system: during the partition, both sides keep accepting writes, and later somebody reconciles. You keep availability and accept temporary divergence. That is the behaviour of Cassandra and DynamoDB in permissive configurations.

And it is worth saying: most real systems are configurable between the two, and the configuration matters more than the product label.

PACELC: the part you use every day#

Daniel Abadi proposed an extension that should be more famous than the original:

If there is a Partition, choose between Availability and Consistency. Else, choose between Latency and Consistency.

The second half is the one that governs your daily life, because partitions are rare and the "else" is the rest of the time.

• Every time you read from a replica instead of the primary, you traded consistency for latency.
• Every time you use asynchronous replication, same thing.
• Every time you put a cache in front, same thing.

No partition involved. Just design decisions that almost never get recorded as decisions, and that produce the most confusing bugs to debug.

The consistency ladder#

Consistency is not yes or no. It is a ladder, from strongest to weakest.

Linearizable. Everything behaves as if there were a single copy and operations happened one at a time, in real clock order. It is the most intuitive and the most expensive: it requires coordination on every operation, which means network round trips.

Sequential. Everyone sees the same order, but that order does not have to match real time.

Causal. If A caused B, everyone sees A before B. Operations with no causal relationship can show up in different orders for different observers. It is a very good balance point, and it is what messaging and feed systems generally want: you never see the answer before the question.

Eventual. If you stop writing, at some point everyone converges. It does not say when. It is the cheapest and the easiest to sell dishonestly.

1. Linearizablemost expensiveAs if there were a single copy, in clock order. Coordination on every operation.
2. Sequentialone orderEveryone sees the same order, and it need not match real time.
3. Causalgood balanceIf A caused B, everyone sees A first. Never the answer before the question.
4. EventualcheapestStop writing and everyone converges. It does not say when.

Session guarantees: the practical shortcut#

Here is the concept that solves most real problems without paying the price of strong consistency.

Instead of guaranteeing a global property of the system, you guarantee properties for a user session:

Read your own writes. The user always sees what they themselves just saved.

Monotonic reads. The user never sees the clock run backwards: if they have already seen newer data, they cannot see the older version afterwards.

Monotonic writes. Their writes are applied in the order they made them.

Why this matters: almost every consistency bug that reaches support is a violation of one of those three. The user saves the profile, gets redirected, the screen reads from the replica, shows the old data, and they save it again.

How to implement "read your own writes"#

Three approaches, from the simplest to the most correct:

1. Primary window. After a write, read from the primary for a few seconds for that user. Flag it in the session or in a cookie. It solves ninety per cent of cases with very little code.
2. Log position. Store the replication log position at write time in the session, and only accept reading from a replica that has already passed that point. More correct, more work, and it requires support from the driver or the proxy.
3. Critical routes always on the primary. An explicit, documented decision for a small set of routes.

The worst approach, which is the most common one, is to do nothing and treat every complaint as an isolated case.

What to ask in the next meeting#

Swap "strong or eventual?" for three better questions:

Which guarantee does the user need to perceive? Almost always the answer is "they need to see their own change".

What is our replica's real lag, at p99, over the last week? If nobody knows, that is the first job.

What happens during a partition, and have we tested it? The honest answer in most teams is "we do not know", and finding out in a controlled test is a lot better than finding out in an incident.

An honest closing#

Strong consistency is not always the right answer, and eventual is not always enough. What separates a good decision from a bad one is not the level you picked, it is having picked deliberately and having written down why.

A half page ADR answering "why we accept eventual reads on this route, and how we mitigate the effect on the user" is worth more than any argument about acronyms.