Tests worth what they cost

Start with the wrong question and you end up with an expensive, useless suite.

The wrong question is "how much of my code is covered?". The right one is "do I have the courage to refactor this?".

If the answer is no, the suite is not doing its job, whatever the percentage says.

Pyramid, trophy, and the argument that does not matter#

The classic pyramid: many unit tests at the base, some integration tests in the middle, few end to end at the top. The logic is cost and speed: the higher you go, the slower and more brittle.

The trophy, from Kent C. Dodds, redistributes: few unit tests, many integration tests, few end to end, and static analysis at the base. The argument is that most real bugs live in the interaction between the pieces.

Who is right depends on what your system does.

If you have dense business logic (price calculation, tax, eligibility rules), the pyramid works: there is a lot of isolated, deterministic behaviour worth pinning down.

If your system is mostly glue between a database, a queue and an API, the trophy works better: testing each class in isolation with mocks proves almost nothing.

The question I prefer to the format argument#

If this test passes, what exactly can I claim about the system?

If the answer is "that this method calls that other one", the test is worth little.

If it is "that an order with an expired coupon is rejected", it is worth a lot.

The attributes of a good test#

Kent Beck listed them, and the list is practical:

Fast. A slow suite stops being run before commits, and feedback goes from seconds to half an hour.

Deterministic. Same code, same result, always.

Isolated. It does not depend on another test or on execution order.

Readable when it fails. The message should say what broke without you opening a debugger.

Sensitive to real bugs and insensitive to refactoring. This is the hardest and the most important.

The doubles, and why mocks are dangerous#

The names are specific and people swap them:

Stub returns a canned answer. You use it to control the input.

Mock verifies interaction: "that method was called once with these arguments".

Fake is a real implementation, simplified: an in-memory repository.

Spy is the real thing, wrapped, recording the calls.

When you verify interaction, you pin down the internal design of the code.

Then you refactor (extract a method, change the order of two calls, swap one dependency for another) and twenty tests go red without a single behaviour having changed.

That teaches the team that tests are an obstacle. And it teaches it correctly, because in that case they are.

The rule: test observable behaviour, not implementation. This goes in, that comes out. Use mocks only at the boundary you do not control: the payment gateway, the email sender, the clock.

When in doubt between a mock and a fake, prefer the fake. A thirty line in-memory repository solves more and breaks less.

1. Stubcontrols the inputReturns a canned answer. It verifies nothing, it just feeds the case.
2. Fakethe default choiceA real implementation, simplified. A thirty line in-memory repository.
3. Spythe real thing, watchedThe actual object, wrapped, recording the calls.
4. Mockpins the designVerifies interaction. Refactoring turns twenty tests red with no behaviour changed.

The symptoms of a brittle test#

If you recognise three of these, your problem is not missing tests, it is too many of the wrong kind:

→ it breaks when you rename a private method

→ it depends on execution order

→ it uses mocks chained three levels deep

→ it verifies string formatting instead of the data

→ it has more setup lines than assertions

→ nobody on the team can explain what it guarantees

Coverage: a map, not a target#

Coverage measures what was executed, never what was verified. A test with no assertions gives you a hundred per cent on the line it runs.

As a target it becomes theatre. Goodhart's law: when a measure becomes a target, it stops being a good measure. With an eighty per cent target, the team covers getters and DTOs (cheap and useless) and leaves the hard rule out.

How to use it well:

→ As a map: look at what sits at zero, especially in critical code.

→ On the diff: require coverage on the new code in the PR, not a global number. The curve rises on its own, in the right place.

→ Branch coverage, not line coverage.

And if you want the honest answer about your suite's quality, there is a better test than any report: mutation testing. The tool sabotages your code (swaps > for >=, flips an if, removes a call) and runs the suite. Whatever stays green was not being verified.

You can do the manual version in two minutes: flip a comparison in your most critical calculation and run. If it passes, the silence is your answer.

A flaky test kills the suite#

A test that passes and fails on the same code is worse than a missing test, because it teaches the team to ignore red.

With a two per cent random failure rate on a suite of five hundred tests, almost every run has a red. In two weeks, "just run it again" became process.

The causes: fixed sleep (swap it for waiting on a condition with a deadline); order dependence; real concurrency; external resources, including the clock, randomness and time zones.

The policy: a flaky test becomes an issue with an owner and a deadline. Not an automatic retry: the retry hides exactly the concurrency bug the test was, correctly, finding.

How to find yours: run the suite twenty times on the same commit, overnight. Three or four tests usually account for eighty per cent of the false reds.

TDD, without religion#

The cycle is red, green, refactor. The main gain is not the test you are left with: it is being forced to use your own interface before implementing it.

Where it shines: business rules with clear inputs and outputs; bug fixes.

Where it gets in the way: exploration, when you do not know the design yet; integrating with a third party system you do not understand yet.

And the least controversial, highest return use, which I would recommend to any team, even the ones that do not do TDD:

Every fixed bug ships with a test that fails before the fix.

It costs fifteen minutes, it proves you understood the cause, and it guarantees that bug does not come back. In three months you have a regression suite built from the mistakes your system actually makes, far more valuable than one built from what somebody imagined might go wrong.

The pipeline#

Closing with the structure that holds all of this up: stages from fastest to slowest. Static checks in seconds. Unit tests within five minutes. Integration with a containerised database within fifteen. End to end and deploy at the end.

Feedback in under ten minutes through the third stage. Above that, people stop running it before committing.

And the green trunk rule: a broken main is a team emergency. Either it is reverted in minutes, or somebody stops and fixes it. Nobody builds on a broken base.