Tests are a cornerstone of modern development, and somewhere along the way we started treating a green suite as proof. It isn’t: it tells you the cases you thought of still pass, not the ones you didn’t. That confusion was always with us — and it matters more now that an AI agent might be the one chasing the green.

Program testing can be used to show the presence of bugs, but never to show their absence!

— Edsger W. Dijkstra

Test-driven development (TDD) is a fine principle: write the test first, let it drive the development. No argument from me. But this post isn’t about the method. “TDD” here is my umbrella label for a belief: that a green suite makes code correct — perform the incantation, ship the green, and nothing can harm you. True believers preach it openly. The teams that shrug, “we don’t really do TDD, but we’re as serious about tests as you can get,” often hold it without the vocabulary — and a serious tester can be free of it entirely. Seriousness isn’t the tell; the belief is. It doesn’t work that way, and it sticks for a reason that has nothing to do with correctness.

A green number is legible to everyone. Engineering judgment is legible only to other engineers.

Full disclosure: I’ve written more than one test harness myself — the latest is tape-six . It goes green like the suites I’m about to distrust. This is a critique from inside the church, not a heckle from the door.

Green means done

The belief has a pedigree. Long ago I read a book on testing — I no longer recall the title or the authors — from the days when the waterfall was all the rage. It recommended, in full seriousness: produce the specs, have one team design and implement the complete test suite — all red at first, since nothing is built yet — then have a different team implement the features, and watch the suite grow greener. All green — we are done. I never saw it done, nor met anyone who has.

But the tenet survived: design the test first; it’s red; implement the feature; the test turns green — done. It is the foundation of TDD and of extreme programming ’s test-first. Yet what I mostly see is the opposite order: developers write the feature, then write a test for it — and when the test comes up red, they fix one or the other, sometimes the code, sometimes the test. The discipline was dropped; the finish line — green means done — is the part everyone kept.

Tests sample, they don’t prove

The loop is familiar: change the code, run the suite, it’s green, nothing broke. Yet the code can still be riddled with bugs — because for almost any wrong implementation, there’s a plausible-looking set of inputs that passes.

const badMul = (a, b) => a + b; // "multiplication"
assert(badMul(0, 0) === 0); // green
assert(badMul(2, 2) === 4); // green

+ agrees with * at the two points we happened to sample. The suite is green; the function is wrong. Tests catch bugs by sampling — and the sample isn’t even random: it’s drawn from the author’s imagination, the cases somebody thought to write down. Picking the edges and the common paths improves the odds, but never covers it all.

A friend once brought over his newest application and demoed it — a specialist in its domain, he flew through every common use case. He left it with us for feedback; the moment he was gone I sat down, clicked around the menus looking for how to create a new document — and it crashed. And again. And again. I called him; he was astonished — he had tested the app thoroughly and used it daily for his own work. On his next visit, watching me crash it live: “Oh, you are doing it completely wrong!” I was: wrong order of actions, actions that made no sense, Help where an input was expected. It didn’t matter. His tests sampled his imagination, and I wasn’t in it.

Where are the edges?

“Sample well” smuggles in an assumption: that we know where the edges are. Some are logical, handed to us by the problem — the sum of an empty array is 0, the boundary where a loop runs zero times. A careful read of the spec turns them up.

The dangerous ones exist only because of how you built the thing. Say you handle x < 0 with one algorithm and x >= 0 with another: nothing in the specification calls 0 special, but you just did. Now you have to check that the two halves meet at the seam — no gap, no double-count, no off-by-one at the border. That edge is invisible from the outside; only someone who has read the implementation knows to test it. A spec-based suite walks past it.

And these edges breed. A cache adds a cold path and a warm one. A fast path for small inputs adds the threshold where it hands off to the slow one. A retry adds the seam where the second attempt must not repeat the side effects of the first. Every such choice mints a boundary the requirements never mentioned — and the boundaries move under refactoring, so the suite that knew them last month is stale this month. It still passes: a test pins the assertions as written, not the intent behind them. “Test the edge cases” assumes a current map of edges that most tooling can’t even see.

Coverage is not correctness

A coverage tool is useful: it shows the lines no test ever runs. Used as a target, though, it turns into a number to chase — every line of our code should be touched by some test — and a green 100% gets read as proof. It sounds airtight.

It isn’t, and the reason is combinatorial: code branches, and each branch sets up values downstream. One if runs two ways. N independent ifs run in 2^N combinations — ten of them in a row are 1024 distinct paths through the same block. Two tests — one running every if true, one running every if false — and the tool reports 100%: every line executed, every branch taken both ways. The other 1022 combinations were never tried.

And the bug that bites is almost always a combination:

let x = new Person();
if (a) x = null;
if (b) x.setParent(y);
// a && b -> null dereference; every other combination is fine

Two tests — a alone, then b alone — execute every line and every branch. Both pass. The bug lives only in a && b, the one combination “100% coverage” never demanded. The metric is green and the code is broken.

And ten ifs is a toy. Real systems carry them by the thousand. Granted, not all ifs are independent, but their combinations are still astronomical: more than any suite will ever sample meaningfully. A team that relies on testing alone ships bugs with the suite’s blessing.

Coverage measures what code ran, not what cases we checked. It is a floor — code with none is certainly under-tested — never a ceiling. Treated as a goal, the number pays out in fully-covered bugs.

Better sampling is still sampling

The toolbox has grown past hand-picked examples. Property-based testing generates inputs against a stated property; fuzzing does it at industrial scale; mutation testing grades the suite itself — it plants bugs on purpose and checks that some assert notices, measuring exactly the gap between ran and checked. Worth knowing, all of them. None changes the arithmetic.

Random inputs are a Monte Carlo method : sample a space too large to enumerate and hope the draws land where it matters. Monte Carlo is the tool of last resort for a reason — it’s expensive and it converges slowly — and a program’s input space is exactly the kind of place where nothing better applies. Generators hedge by favoring the usual suspects — 0, -1, the empty array, min and max — which works exactly when the edge sits on a usual suspect. An implementation’s seam sits wherever the implementation put it: at 1024, at a cache’s capacity, at sqrt(pi)/4 if the algorithm demands it. For those the draw is as blind as any other.

Coverage-guided fuzzers steer the sampling instead of hoping, and steering helps — when the feedback cooperates. The samples stay expensive, the hit stays statistical — and somewhere along the way we stopped testing and started collecting statistics. None of this is a dismissal; it’s a price tag. Every method works within its limits — know them.

Which is the real point: testing is hard. “We have ten thousand asserts!” And? Do they test the right things? A suite is designed, not accumulated — and no tool does the aiming for us.

And some of what a real system does is random by nature: how requests arrive, how mutations interleave, what users do next. No suite guarantees a catch there — on the large projects I’ve worked on we always had a percentage of tests failing randomly. Note how the well-known systems tamed nondeterminism: not by more testing, not by Monte Carlo, not by fuzzing — by a solid theoretical foundation. That’s why databases have transactions and every DBA recites ACID.

Snapshot testing is an oracle nobody checked

There’s a problem upstream of all this sampling: where do the expected values come from? In simple cases we work the answer out by hand. In complex ones we cheat , and not in the honorable sense: run what we have and record the output as the “expected” value, or press a previous version of the program into service as the oracle. The technique has a name, snapshot testing, and major frameworks support it. (Testing against a genuinely independent reference implementation is a different game; there, at least, the oracle answers to its own spec.)

It does catch unintended drift: the output changed, come look. But it never establishes that the recorded answer was right to begin with. A snapshot of a bug is a green test that pins the bug in place. It automates “it still does what it did,” not “it does the right thing.”

The suite is not free

Everything above is about what a green suite proves. Now, what it costs. The library only grows: tests get added for good reasons and, in my experience, almost never retired. A test written to pin a dependency’s bug or an algorithm’s seam outlives the bug, the dependency, and the algorithm — the runtime ships the fix, the workaround test stays, green and testing nothing. Auditing a suite for dead weight is real work, and no manager budgets for it.

Catching regressions is the suite’s legitimate job — but every run pays for the whole library. A confined change with a tiny blast radius still buys the full run; on a serious project, with related projects chiming in, that’s forty minutes and counting. Tooling to trim the bill exists — run only the affected tests, queue and bisect the merges — and plenty of shops never wire it up. I’ve watched them cope by batching changes and running the suite on a schedule instead — saving machine time by making everything else worse: the lag grows, and when a batch goes red, whose change broke it?

Re-running isn’t the sin — while the suite is fast, it’s a virtue. Development runs on flow, the state where edit and check chase each other without a seam, and fast tests sustain it: a cursory check on every edit, watch modes re-running on every save. Compiled languages taught this lesson long ago — a long compile broke the flow at every edit. Coming from C++, I found Python and JavaScript running code immediately a gift; a forty-minute suite takes the gift back. The library grows until the loop that was supposed to protect development speed is the thing that kills it.

And what does the run buy? It exercises a test environment, not production: the network, the data, the files are stand-ins, and some bugs live only in the difference — every major project I’ve worked on was bitten by that gap, regularly. One shop I worked at kept three test environments — dev, unit, integration — each different, each catching a little more; the gap to production outlived them all.

The cult of 100%

The coverage number’s hollowness as proof would be a footnote if it weren’t worshipped. But it is. It has become the KPI of quality — managers quote it, open-source projects wear a “100% coverage” badge like a medal, and plenty of teams gate the merge on it: no PR lands until the number reads 100%.

Outside the true believers, developers rarely preach the belief. Most keep a healthy doubt — nobody who has shipped software thinks a battery of tests, however huge, catches every bug. A change can land with the whole suite green and carry a brand-new bug the suite is blind to; old latent ones sit undetected for years; user complaints arrive as the real evidence, regularly. The worship is institutional: the person doubts, the process believes. The gate, the badge, the KPI act on green-as-proof even where no one would defend it out loud — and the manager quoting the number is as trapped as anyone, because the metric is the only view of quality legible to them.

I’ve worked under that gate, and the last few percent are where it turns absurd. Chasing 100% means covering the paths that are hard to trigger: the disk that fails mid-write, the allocation that returns null, the error branch for a condition the harness has no way to produce. Reaching them takes mocking and fault injection, all to exercise code that is usually three trivial lines. The payback is a green number and nothing else.

And there are cheaper ways to the same assurance. A static analyzer can show the branch isn’t dead code, and reading its three lines confirms they do what they say — no manufactured failure required. That assumes the reader got it right — a safe bet for one careful engineer, a shakier one across a whole team — and covering that gap is the gate’s one honest function: a blunt proxy for did someone competent look. So why does the ritual win — days of mocking over minutes of reading? Because the metric you can manage beats the verification you can’t.

AI takes green as gospel

All of this gets sharper the moment an AI agent is doing the work. An agent’s feedback loop is the suite: it writes code, runs the tests, the linter, the style checks, sees green, and reports success. Green is the strongest signal it gets, so green is what it optimizes for.

Reasoning is the work we’d most want a capable agent on, and there are two good ways to spend it. An agent can review: read the argument the code makes, flag where it breaks — done once, when the code is written. Or an agent can write the code and reason as it goes — invariants, proof, correct by construction — with us as the second set of eyes. Either way the reasoning is paid for once, at writing time.

Whether agents are good enough at any of this yet is debatable; where we aimed them isn’t. We pointed them at the suite instead — the ever-larger library, re-run on every cough, forever — and an agent lives in that loop full-time, every lap at suite speed. We built the most tireless reviewer we’ve ever had and put it on the treadmill.

I have watched agents “fix” a failing test by editing the test until it passes — enshrining the bug. I have watched them assert against the wrong thing, usually an oversight, and the suite goes green all the same. A person at least can feel the itch that something is off; an agent takes the checkmark at face value and moves on. Hand the keys to a process that treats green as gospel, and every weakness above compounds.

Summary

None of this makes tests useless; far from it. The mistake is treating them as proof — the cargo-cult reflex that performing the tests is what makes code correct. Only reasoning about the code does. Tests sample the space of behaviors; they prove nothing about the whole of it, and coverage measures the sampling effort, not the result. And the suite doing the sampling is not free: it has to be designed, pruned, and kept fast, or it kills the very speed it was built to protect.

The next post takes it up: loop and if invariants, and the lightweight formal proofs they buy. Tests catch where our reasoning slipped; invariants prove the parts we reasoned through. We want both.