Latency, throughput and percentiles: the guide that settles the argument

There is a sentence that shows up in every technical meeting and means nothing: "the system is slow".

Slow for whom? At what moment? Compared to what? Until those questions get answered with numbers, the conversation is about feelings. And you cannot optimise a feeling.

Latency is not throughput#

Latency is how long one request takes. Measured in milliseconds. It is one person's experience.

Throughput is how many requests fit per second. Measured in QPS. It is the system's capacity.

The motorway analogy works: latency is how long your car takes from London to Manchester, throughput is how many cars pass the toll booth per hour. You can double throughput by building more lanes, and your trip still takes exactly as long.

Why this matters in practice: when somebody reports slowness, the first question is which of the two changed.

• Latency went up, throughput flat → some dependency got slower. Look outside your code.
• Throughput hit a ceiling and latency shot up with it → you saturated. There is a queue growing somewhere: load balancer, connection pool, task executor.

Two completely different diagnoses, and telling them apart takes thirty seconds if you have the right graphs.

Little's Law#

There is a relationship between the two that settles half the capacity arguments:

requests in flight = throughput × latency

A hundred requests per second, two hundred milliseconds each, gives twenty requests alive at once.

That number tells you how many threads you need, how many database connections, how much of any resource that stays busy for the duration of the request. It is a multiplication: not modelling, not guessing.

And it exposes a trap: if latency doubles, the number of in-flight requests doubles, at the same traffic. That is why a slow dependency drains your thread pool on a day when volume did not change at all.

The average lies. The percentile does not.#

If I tell you my API's average latency is 100ms, you know almost nothing.

Picture ten requests: nine took 10ms and one took 900ms.

d = [10]*9 + [900]
average = sum(d) / len(d)   # 99.0  -> looks great

The average comes out at 99ms. But 10% of users had a horrible experience, and the dashboard is green.

Percentiles fix that. You sort all the latencies, smallest to largest, and look at where each cut lands.

Tail amplification#

Now the part that changes your architecture.

Suppose your page makes ten internal calls to build the response, and each service has a p99 of 1 second. What are the odds the whole page is slow?

It is not 1%. It is the chance that at least one of the ten lands in the tail: roughly 10%.

That is called tail amplification, and the consequence is harsh: the more you split the system into services, the more each one's individual p99 turns into the end user's p90.

In distributed architecture, component p99 is a collective budget, not an individual metric. Every new service on the critical path spends part of that budget.

What to do about the tail#

The typical causes, worth hunting in this order:

1. GC pauses: periodic spikes with no correlation to traffic.
2. Lock contention: gets worse with concurrency, does not improve with more CPU.
3. Cold cache: after a deploy or a mass expiry.
4. Noisy neighbour: another process on the same machine.
5. TCP retransmission: packet loss; TCP shrinks the window and latency jumps.
6. Queueing: anywhere along the path, and the most common of the six.

And there is a direct technique against the tail, popularised by Google, called hedging: if the request has not answered by the p95, you fire a second one at another replica and use whichever response arrives first. You pay about 5% extra traffic and cut the tail substantially.

How to start today#

Three concrete steps, in order:

1. Replace averages with percentiles on every dashboard. If your metrics system only keeps averages, it is hiding the problem from you.
2. Measure p50, p95 and p99 per endpoint, not just the service aggregate. The aggregate hides the bad endpoint behind the popular fast one.
3. Compute in-flight requests with Little's Law and compare with your pool sizes. Much larger than needed means contention you are paying for nothing; smaller means queueing.

Once that is done, "the system is slow" stops being an opinion and becomes a sentence with a number, an endpoint and a percentile.

And then you can fix it.