Caching: the four traps nobody anticipates

Phil Karlton said there are only two hard things in computer science: cache invalidation and naming things. The joke aged well because the first half is still true.

Caching is storing an expensive result so you do not recompute it. The idea is trivial. What is not trivial is deciding when that result stopped being valid, and what happens to your system the moment many of them stop being valid at once.

Where to cache#

From closest to the user to furthest away:

1. BrowserfreeDriven by an HTTP header. The most underused layer.
2. CDNedgeServes close to the user, with the s-maxage directive.
3. Application localnanosecondsIn process memory. Each instance has its own, and they diverge.
4. Distributedone round tripRedis, Memcached. Shared and consistent across instances.
5. Databasenot yoursBuffer pool and page cache. Explains the faster second run.

Browser. Free, fast, and you control it with an HTTP header. It is the most underused layer.

CDN. Serves content from the edge, close to the user. Also header driven, with the s-maxage directive.

Application local cache. In memory, in the process itself. Nanosecond latency. The problem: each instance has its own, and they diverge.

Distributed cache. Redis, Memcached. Shared across instances, consistent between them, at the cost of a network round trip.

Database cache. Buffer pool, operating system page cache. You do not control it directly, but it explains why the second run of the same query is so much faster.

The three strategies#

Cache-aside is the most common: the application looks in the cache; on a miss it goes to the database and writes to the cache. Simple, you control everything, and the cache can go down without taking the application with it.

Write-through writes to the cache and the database at the same time. More consistent, slower writes.

Write-behind writes to the cache and to the database later, in batches. Fast and risky: if it dies before flushing, you lost writes.

For most cases, cache-aside is the right answer. The other two solve specific problems and bring failure modes of their own.

Trap 1: stale data#

The TTL is the numeric expression of your tolerance for being wrong.

There is no correct TTL in the abstract. There is the answer to: "how long can this data be out of date before anybody cares?".

Product price: seconds. Category name: hours. Feature configuration: minutes. The logged-in user's profile: that is the hard one, because the user notices immediately, and there the answer is to invalidate on write, not to trust the TTL.

The common mistake is picking a single TTL for everything, usually five minutes, because that is what the documentation example used.

Trap 2: cache stampede on expiry#

You populated the cache during a deploy, with a one hour TTL. One hour later, everything expires at the same instant, and all the traffic hits the database at once.

The cause is synchronisation. The fix is jitter: add a random variation to the TTL, say between 50 and 70 minutes instead of exactly 60. That spreads the expiries out.

It is the same idea as jitter in retries, and for the same reason: synchronised events create spikes the system cannot take, even with plenty of average capacity.

Trap 3: the herd#

A popular key expires. A thousand simultaneous requests discover that in the same millisecond and all of them go recompute the same thing.

The name is thundering herd, and the solution is to let only one of them recompute:

• A lock per key: whoever takes the lock recomputes; the others wait or serve the old value.
• Or early recomputation: when expiry is close, one randomly chosen request refreshes in the background while the others keep serving the current value.

The second is better, because nobody waits.

Trap 4: negative caching#

Somebody queries an id that does not exist. You find nothing and store nothing. The next query for the same missing id hits the database again.

If that arrives in volume (and it does, because that is how scrapers and automated tests behave), you have a route that bypasses the cache entirely.

Store the negative result too, with a short TTL. And if the volume is genuinely large, a Bloom filter in front answers "definitely does not exist" without even touching the cache.

The cache as a critical dependency#

A well implemented cache-aside degrades: without the cache, everything gets slower, and it keeps working. That requires the path to the database to still exist and your database to survive the load without the cache, even if that means shedding some.

The number that matters#

Monitor hit rate per key or per prefix, not just the global one.

Below eighty per cent on a read cache, one of two things is wrong: the key is too granular, or the TTL is too short for the access pattern. In both cases you are paying the complexity and the inconsistency without getting the benefit.

And monitor the invalidation rate as well. If it is high, you are spending more time invalidating than serving.

Practical summary#

A cache trades freshness for speed. The engineering work is not switching the cache on; it is deciding explicitly how much freshness you are willing to lose, per key, and then defending the system against the four known ways that goes wrong.