We spent months building a code graph. Then we asked whether it helps our reviewer find bugs. It does not.
Eight commits improved. Eight got worse. The test that measures exactly that returned p=0.803.
We wrote the prediction down first
A separate lane had measured retrieval on third-party code at 80.1 percent recall@4, against a 5.5 percent negative control, n=161. Retrieval finds the right code. The reviewer was still finding about a third of injected defects.
So we wrote this down before the run:
The one-third finding-recall is not "we cannot locate the right code." It is "we do not recognise the defect once we are looking at it."
That predicts something specific. Hand the finder a much richer view of exactly the right code and it should buy nothing.
Arm 4 tested it. It bought nothing.
The setup
85 real merged commits from goose and litellm, repositories we did not write. One synthetic defect each: 31 equality flips, 47 connective flips, 7 boundary flips.
Every selection rule ran before any model call. At least 180 days old. Code files only. Six files or
fewer. No later commit matching fix|revert|hotfix|bug|patch|regress touching the same paths.
That last rule reads at most 40 later touches per path. A capped read fails open, so we say so.
Four arms, same 85 commits
| arm | context given to the finder | recall |
|---|---|---|
| 1 · false alarm | none, no defect injected | 2 to 5% across three samples |
| 2 · ungrounded | none | 33% |
| 3 · grounded | commit-local graph, median 577 bytes | 38% |
| 4 · full repository | whole repo at the commit's revision, median 4,115 bytes | 33% |
Arms 2 and 3 each replicated across two runs, moving by one commit out of 85.
The paired test
Every arm ran the same 85 commits. Wilson intervals are conservative on paired data and can hide an effect. McNemar throws away the commits both arms agree on.
| comparison | both found | only A | only B | discordant | p |
|---|---|---|---|---|---|
| ungrounded vs commit-local | 21 | 7 | 11 | 18 | 0.480 |
| ungrounded vs full repository | 20 | 8 | 8 | 16 | 0.803 |
| commit-local vs full repository | 21 | 11 | 7 | 18 | 0.480 |
Arm 3's five-point gain is a net of four commits out of 85. It does not survive the paired test either.
Per defect class the direction is not even consistent. Equality flips went 10/31 to 14/31. Connective flips went 16/47 to 15/47, which is worse. Boundary flips went 1/7 to 2/7, which is seven samples and gets reported rather than interpreted.
What is falsified
Grounding as this pipeline builds it. build_anchors over a code graph, injected into the finder prompt,
improving recall on diff-level defect detection. On 85 real third-party commits it does not, at either
graph size.
Not grounding in general. Retrieval works. The gate is a different mechanism with its own evidence. What failed is anchors-as-finder-context for recall.
One defect family. Find-only. Two repositories.
The finder is the ceiling. Richer context does not raise a ceiling that was never about context.
Why publish it
We built a bigger graph because it was obviously going to help. It obviously did not.
Every team building on retrieval right now is about to spend a quarter on the same intuition. The measurement cost us a day.