Here is something that happened to us, and some version of it has happened to you.
We had a check guarding an important behaviour. It ran on every change. It had a good name that described exactly what it was for. It had been green for months, and one day somebody looked at it properly and realised it was asserting on the response when the thing it existed to catch lived in the request.
It had never been able to fail. Not once. Months of green over a hole.
The three kinds of broken instrument
The one everybody catches is the test that errors. It goes red, somebody fixes it before lunch, nothing is learned and nothing is lost.
The second kind can’t fail. It runs, it reports, it is pointed slightly away from its subject. You won’t find this one by reading the output, because the output is perfect.
The third kind is the worst and almost nobody looks for it. It covers part of the contract correctly and says nothing about the rest, so its green reads as a verdict on the whole thing. We found one that enforced two clauses of a three-clause rule. It was genuinely working. It was also silent about a third of what it claimed to guard, and that silence rendered exactly like a pass.
What we do about it now
Two questions before anyone trusts a green run.
Which half does this cover? Say it out loud. If you cannot describe what the check does not look at, you do not know what it looks at either.
Can it fail? Break the subject on purpose and watch. If it stays green over a broken thing, you never had a check, you had a decoration that happened to run.
This takes a minute. People skip it because a checkmark is right there and reading it is free.
Why it shapes the product
An agent that writes code and runs your suite has produced a claim. If we hand that claim to you wearing the costume of a verdict, we have built a faster way to be confidently wrong, and the industry is not short of those.
So when we tell you something passed, we try to tell you what it passed, and what nobody checked.