The five resilience patterns that stop a cascading failure

The incident always has the same shape.

A secondary dependency gets slow. It does not go down, it gets slow. Every request that calls it holds a thread waiting. The threads run out. Now there is no thread left even for the main flow, which had nothing to do with the problem.

Ninety seconds between "recommendations are slow" and "checkout is down".

What separates a system that fails well from one that fails badly is five patterns. None of them is complicated. All of them are easy to forget.

1. Timeoutmost forgottenA little above the dependency's p99. Without one, a call can last forever.
2. Retry with backoff and jitteronly what is safeWithout jitter, a thousand clients come back together and you built a wave.
3. Circuit breakerfail fastDead dependency: reject immediately and use plan B.
4. Bulkheadseparate poolA secondary feature dies inside its own pool.
5. Load sheddingby priorityBetter to serve eighty per cent well than one hundred per cent badly.

Pattern 1: Timeout#

The cheapest and the most forgotten. A call without a timeout is a call that can last forever, holding a thread and a connection.

And the default in many libraries is thirty seconds, or infinite.

How to pick the value: look at the dependency's p99 and set it a little above. If the p99 is two hundred milliseconds, a one second timeout is generous. A timeout far above the p99 protects nobody; it only postpones the problem.

The hierarchy has to make sense: the caller's timeout has to be larger than the callee's, otherwise the inner one never fires and you lose the information about which layer failed.

Add up the whole path and compare it with the deadline your user tolerates. If the sum of the timeouts on the critical path goes past ten seconds and your user gives up at three, your configuration is fiction.

Pattern 2: Retry with backoff and jitter#

Retry looks harmless and is one of the most efficient ways to take down your own system.

Only retry what is safe to retry. GET and DELETE, no problem. A POST that creates an order, only with an idempotency key. Without one, this is how you double charge someone.

Exponential backoff: wait 100ms, then 200, then 400. Without it you hammer a service that is already suffering.

Jitter is the part almost everyone forgets, and it is the most important one. If a thousand clients failed in the same second and all of them wait exactly the same interval, they all come back together. You just built a synchronised wave.

Jitter means randomising: wait a random value between zero and the computed interval. One line of code, and it is the difference between recovering and being stuck in a storm cycle.

Small maximum. Three attempts.

Pattern 3: Circuit breaker#

A breaker switch. It counts a dependency's failures; past a threshold it opens and starts rejecting immediately, without even trying. After an interval it goes half open and lets a single test request through. If that works it closes; if not, it opens again.

Why it is essential: when a dependency is dead, insisting only makes things worse. You hold your own resources waiting for something that is not going to answer.

Failing fast gives your threads back and lets you use plan B: stale cache, partial response, honest message.

The parameters that matter: the failure threshold should be a rate, not an absolute count, so it does not open because of ten errors in a million requests. And the open period should be short enough to recover quickly once the dependency comes back.

The common mistake: a circuit breaker with no plan B. It opens, it rejects, and the user gets an error anyway, just faster. That is still better than hanging, but half the value is in the fallback.

Pattern 4: Bulkhead#

The name comes from the watertight compartments of a ship. One section floods and the ship stays afloat.

The idea: separate resource pools per dependency. If the integration with the recommendation service has its own pool of ten connections and ten threads, it can die entirely inside those ten slots without consuming the resources checkout needs.

Without a bulkhead, one slow secondary feature eats every thread and takes the main one down with it. That is literally the mechanism behind most cascading failures.

How to implement it: a dedicated thread pool per dependency, or a semaphore limiting concurrency per external call. The second is lighter and solves most cases.

How to test it: put a proxy in front of a secondary dependency and add thirty seconds of artificial latency. If the whole service goes down, you do not have a bulkhead, and you just found that out in a controlled environment.

Pattern 5: Load shedding and graceful degradation#

When you are past capacity, reject explicitly with a 503 instead of accepting everything and degrading for everyone.

It is counterintuitive and it is correct: better to serve eighty per cent well than to serve one hundred per cent badly. A system that accepts everything and takes longer and longer collapses and does not come back on its own.

And shedding should be by priority: drop the report before the payment. That requires your requests to carry some notion of importance. That is a product decision.

Alongside it comes graceful degradation: design in advance which feature is allowed to disappear. Synonym search goes and plain search stays. Recommendations become a fixed list. Calculated shipping becomes an estimate.

That is a decision made in peacetime, not during the incident. And it is the thing that most separates a mature team from a team that fights fires.

The checklist#

List every external call your service makes. For each one:

1. Does it have a timeout? What value, and where did it come from?
2. Does it retry? Is that safe? Does it have backoff with jitter? Is it at a single level?
3. Does it have a circuit breaker? Is there a plan B when it opens?
4. Does it have a separate pool, or does it share with the critical path?
5. What does the user see when it is down?

I would bet at least one of those calls has no timeout at all, or is still on the library's thirty second default.