Why your service degrades over time

The pattern is familiar: the service comes up fine after the deploy. On the second day it is a little slower. On the fourth, somebody restarts it. And the cycle begins again.

"Restarting fixes it" is a diagnosis, not a solution, and it points at three possible causes.

Cause 1: leaks#

A leak is holding on to a resource you no longer need. There are three kinds, and all of them share the same temporal symptom.

Memory leak. You keep references to objects that should have died. In a garbage collected language, that is not a forgotten free: it is a live reference.

The undisputed champion: an unbounded cache. A Map that only grows, populated on every request, with no eviction policy. It looks like a cache; it is a leak with a nice name.

Runner up: listeners and callbacks registered and never removed. Third: ThreadLocal in a thread pool, where the thread returns to the pool still carrying the value.

Thread leak. You create threads or executors and never shut them down. The thread count grows monotonically. Final symptom: failure to create a thread, or slowness from excessive context switching.

Descriptor leak. A file, socket or connection opened and not closed, usually on an error path that does not go through the closing block. Final symptom: "too many open files".

How to tell them apart: track three numbers over a day: resident memory, thread count, descriptor count. Whichever grows and never falls back is your leak.

ps -o rss= -p <pid>          # resident memory
ls /proc/<pid>/task | wc -l   # threads
ls /proc/<pid>/fd | wc -l     # descriptors

Cause 2: contention#

Contention is threads fighting over the same protected resource.

The signal is distinctive and easy to recognise: you add CPU and throughput does not rise, or it falls.

That happens because the useful work is serialised at one point. More threads means more people queueing for the same lock, more context switches, and more time spent coordinating than working.

The Universal Scalability Law formalises it: on top of Amdahl's sequential fraction, there is a coordination cost that grows with the square of the number of participants. At some point, adding capacity makes performance worse.

How to measure it: do not look only at CPU time. Look at time spent waiting on locks. Most modern profilers have a mode for this; on the JVM, it is the monitor contention profiler.

And false sharing, which is invisible contention: two independent variables land on the same 64 byte cache line. Threads on different cores touch different variables, but the processor invalidates the whole line every time.

You pay for synchronisation without having written a single lock. The fix is padding: filling with space so the variables sit on different lines. It is a rare problem in application code and a common one in high throughput data structures.

Cause 3: the garbage collector#

The generational collector starts from an empirical observation: most objects die young.

So it splits memory into a young and an old generation. It collects the young one frequently, and that is cheap, because most of it is already dead and only the survivors need copying. An object that survives several collections is promoted to the old generation, which is collected rarely.

What hurts is the pause. Modern collectors (G1, ZGC, Shenandoah) do almost all their work in parallel with the application, and pauses have dropped to the millisecond range.

But they still exist, and they still show up in your p99 as periodic spikes with no apparent cause on the traffic graph.

What to do, in order of effectiveness#

1. Allocate less. Most of the GC problem is unnecessary garbage created in a hot loop. String concatenation inside a loop, intermediate objects in a stream, boxing of primitives. An allocation profiler shows this in minutes.

2. Size the heap with room to spare. A tight heap collects constantly. An oversized heap increases the duration of each full collection. There is a sweet spot, and you find it by measuring.

3. Pick the collector by your goal. Throughput or latency: collectors are optimised for one or the other, and the default may not be what you want.

4. Monitor total pause time, not the number of collections. A hundred one millisecond collections beats two half second ones.

How to tell the three apart#

They have different signatures, and that is what makes the diagnosis fast:

Leak: monotonic degradation over hours or days. Restarting fixes it. Memory, threads or descriptors growing without falling back.

Contention: degradation correlated with concurrency, not with time. Worse at peak, better in the trough. Adding CPU does not help.

GC: periodic spikes in the p99, with no correlation to traffic. The p50 stays excellent. It shows up in the GC logs.

If you do not know which one it is, start with the three leak counters, which is the cheapest to rule out.

The most common case of all#

In my experience, the champion is the unbounded cache.

Somebody adds a static Map to avoid a repeated query. It works beautifully. Nobody adds a size limit or an expiry, because "there are only a few keys".

Six months later, the keys include the request identifier, and the map has ten million entries.

The fix is trivial: use a structure with a bound and an eviction policy, something like Caffeine on the JVM or a simple LRU. The hard part is finding it, and the way to find it is a heap dump comparing two moments in time.

The habit that prevents it#

One cheap practice that avoids most of this: a soak test.

Run the system's normal load for a few hours (not peak load, normal load) and track memory, threads and descriptors.

If any of the three grows monotonically over four hours, you have a leak, and you just found it before production did.

It is the kind of test almost nobody runs and that pays for itself the first time it catches something.