Event loop or thread per request: how to choose

There is a recurring argument about which concurrency model is better, and it is usually conducted as an aesthetic preference. It is not. The answer depends on one objective question: is your load dominated by waiting or by computing?

Where the problem comes from#

Blocking I/O is the simple model: you call read, and the thread stops until the data arrives. Easy to write, easy to debug, the error stack makes sense.

The cost is that every connection needs a thread sitting there waiting.

With a hundred connections, fine. With ten thousand, you have ten thousand threads, each with its reserved stack, and the operating system scheduler spending significant time just switching context.

That is literally the problem that became known as C10K, in the early 2000s: how to serve ten thousand simultaneous connections on a single machine.

The solution was to invert the logic. Instead of one thread per connection asking "has it arrived yet?", you register every connection with a kernel mechanism (epoll on Linux, kqueue on BSD and macOS) and ask a single question: "which of these ten thousand are ready right now?".

The kernel returns the list. One thread serves thousands of connections, with nobody parked.

That is what sits underneath Node's event loop, Python's asyncio, Netty on the JVM, Go's runtime. Each wraps it differently; underneath it is the same idea.

Thread per request: what you gain and lose#

You gain: linear code. You read it top to bottom. The debugger works. The exception stack shows the real path. Blocking libraries work with no adaptation.

You lose: scale limited by the number of threads.

And there is a calculation that gives you that number, Little's Law:

busy threads = throughput × latency

A thousand requests per second at a hundred milliseconds each = a hundred threads busy all the time. Comfortable.

If latency rises to one second (because a dependency degraded), you need a thousand. And that is where it starts to hurt: memory, context switching, and the pool probably exhausts before that.

Notice what this calculation reveals: you do not need more traffic to exhaust the pool. Latency rising is enough. That is why a slow dependency takes down a service whose traffic did not change.

Event loop: what you gain and lose#

You gain: it scales beautifully for I/O bound load. Thousands of simultaneous connections with one or a few threads, low memory, no context switching cost.

You lose: one specific and brutal failure mode.

If you do anything CPU-heavy inside the loop, you block everything, for every client. It is not one slow request: it is the whole server stopped.

That is called event loop lag, and in Node one badly placed hash function or heavy serialisation takes down the latency of the entire service. It is the metric you need to monitor on those platforms, and almost nobody monitors it.

And the code is harder: nested callbacks, async/await everywhere, an error stack that does not tell the story, and the requirement that every library in the chain be non-blocking. A single blocking call hidden in a dependency defeats the model.

The decision rule#

Load dominated by waiting (calling the database, calling an API, reading a file, serving websockets): the event loop or lightweight threads win easily. The CPU is idle anyway; what you want is nobody parked.

Load dominated by computing (processing images, compressing, encrypting, compiling, training): you need real parallelism. More cores and a thread or process pool. An event loop here is counterproductive.

Mixed load, which is the common case: an event loop for I/O, with a separate pool for the CPU tasks. That is exactly what Node's worker threads and Python's run_in_executor exist for.

What changed in recent years#

The argument lost much of its relevance because of lightweight threads.

Goroutines (Go) and virtual threads (Java 21+) give you the ergonomics of linear code at the cost of an event loop. The runtime multiplexes thousands of lightweight threads over a few operating system threads, and when one blocks on I/O, the runtime swaps in another automatically.

You write simple blocking code and get event loop scale.

If you are starting a project today on one of those platforms, that is the default choice, and the argument is over.

One caveat: lightweight threads do not solve CPU-bound load. If your work is computation, you are still limited by the number of cores, and you still need to think about real parallelism.

What to measure to decide#

If you are unsure about your own system, three numbers settle it:

1. CPU utilisation during peak. If it is low and the service is slow, you are wait bound, and an event loop or lightweight threads will help.

2. Busy threads versus pool size. Compute it with Little's Law and compare with what is configured. If the pool lives exhausted, you have the classic problem.

3. Time spent waiting versus computing. A profile settles it. If ninety per cent is waiting, the answer is clear.

The mistake that crosses both models#

Whichever model you pick, there is one mistake that shows up in both: not limiting concurrency towards the outside world.

An event loop that fires ten thousand simultaneous requests at an API that can take a hundred will take down the API, and then itself, when the ten thousand error responses arrive together.

The model gives you concurrency capacity. It does not give you permission to use all of it. Semaphores, bulkheads and explicit limits are still necessary.