Green tests, broken software: when the suite is lying to you

The comforting lie of a green test suite is that it means the software works. It does not. It means the things you thought to check, checked out. Those are different, and the gap between them is where the embarrassing bugs live.

Here are two from a large .NET codebase where every test was green and the feature was broken in production anyway.

The fixture that proved nothing#

Four separate outbound integrations, all quietly dead. The pattern was identical in each: the C# response models had no explicit JSON property names on them, so they bound with the default camelCase policy. The real APIs, of course, return snake_case: project_id, is_completed, access_token, expires_in. Every one of those fields silently deserialised to its default value and nobody threw an error until something downstream tripped over the emptiness.

The results were exactly as bad as that sounds. One sync parsed an unbound date string, hit DateTime.Parse(""), threw, and reported “failed” forever. An OAuth refresh never bound the new access_token, so it happily kept the expired one and set the new expiry to “now.”

For a user that meant a feature that looked connected and did nothing. Syncs that reported failure forever, a login that clung to an expired token, and no error loud enough for anyone to notice until they went looking for the missing data.

And the whole time, the unit tests were green. Why? Because the tests fed the models hand-written JSON fixtures using the C# property names, in camelCase. The fixtures matched the code perfectly. They just did not match reality. The test was checking that the model could deserialise a payload the real API never sends.

A deserialisation test written in your own naming convention proves your code agrees with itself. It says nothing about whether it agrees with the API.

The fix in the code was to make the JSON naming explicit at the boundary instead of leaning on a global default. The fix in the tests was more important: pin them to real captured wire payloads, snake_case and all. A fixture that you wrote to match your code will always pass. A fixture captured from the actual service is the only one that can fail for the right reason.

The suite that phoned home#

The other one was hiding in the test harness itself. The integration tests booted the application using its real startup configuration. That sounds responsible until you notice what “real configuration” pulled in: it connected to the actual developer database, spun up the background job server, and, because a health check exercised the AI provider, made live calls to a third-party LLM API using real keys on every single test run.

So the suite was slow, occasionally flaky, and quietly billing an external API to run unit-ish tests. None of that showed up as a failure. It just showed up as sixty-nine seconds and a vague sense that the tests were “a bit heavy.”

Gating the job server behind a flag and blanking the AI keys in the test factory took the suite from 69 seconds to 9, and from 99 passing to 181, because tests stopped blocking on network round-trips to services they had no business calling.

The tell, in hindsight, is that a genuinely isolated test suite should be boring and instant. If your tests are slow, it is worth asking what real I/O they are secretly doing, because “slow tests” and “tests touching production dependencies” are very often the same sentence.

What both have in common#

Neither of these was a logic bug. The code, given the inputs the tests provided, did exactly the right thing. The bug was in the inputs and the isolation: a fixture that did not look like reality, and a harness that reached out into it.

A test is only as honest as two things: the inputs you feed it, and what you let it touch. Get either wrong and you get the worst outcome in testing, which is not a red build. It is a green one you believed.