Code-retrieval benchmark: coding-v1¶

The answer¶

No measurable correctness gain. Every arm's median correctness is 1.00 — grep, graphify, codegraph and hybrid alike. What differs is what asking costs.

Read the claude column only¶

It is the only agent whose tool surface is actually enforced, so it is the only one where "the grep arm" means the agent could not reach a graph tool rather than merely wasn't configured with one.

What the agent does when free to choose¶

Given every tool and no instruction, it reaches for text search 91% of the time. But a single rate is the wrong summary: of 16 questions, 4 ever elicit a graph call and 12 never do. That is a near-deterministic policy, not a low sampling rate.

The honest framing¶

A null result on 16 questions — not proof that code graphs are worthless. One question still runs against the graph arms, two now run in their favour, and the earlier headline finding was withdrawn when it turned out to measure a broken index rather than a backend.

What changed, and why the old finding was wrong¶

For four runs this page reported CodeGraph failing every L2 cell and called it "the finding — nothing else is that clean". It was an artefact: the index was serving the wrong tree, so the arm was quietly degraded to Read, with a broken tool attached. Reading files is expensive, which is also where its apparent cost penalty came from.

With the index repaired, that arm takes all of those cells and its median tool calls on L2 drop from 12 to 2 — it had been flailing against a tool returning nothing. The cleanest result on the page was the cleanest measurement of a defect.

Two lessons travel with that. A benchmark's most striking finding deserves the most suspicion, and a cost comparison against a broken backend measures the breakage.

Reading the arms¶

CodeGraph now runs at 1.05× grep's tokens and 0.93× its latency — marginally faster than grep at roughly the same cost to run. Graphify remains the expensive one at 2.0× tokens and 1.8× latency.

The hybrid arm is the one to read the others against, because it is the only one shaped like production: every tool available, free choice. It is now the fastest arm on the page.

What the agent actually reaches for¶

Across 48 cells hybrid made 227 tool calls, 20 of them graph queries, with 18 of 48 cells consulting a graph at all. Pooled across four runs with a byte-identical tool surface — 248 cells, 1,084 tool calls — 17 reach a graph tool: 1.57%, 95% CI [0.1%, 3.0%] when the interval is clustered on the question rather than computed over calls.

That clustering matters. Computing the interval over calls treats 1,084 correlated observations as independent and produces a confidently wrong band.

The surviving finding¶

A1 is the only result left running against the graph arms: CodeGraph misses all three of its cells, Graphify and hybrid one of three, grep none. The answer is a six-line YAML comment, and neither backend indexes YAML — a fact about what a code index contains, not about how well it searches.

It is trustworthy precisely because the index repair left it untouched, while erasing L2.

What it does not show¶

Sixteen questions, one repository, one model. The hypothesis the set was built to test — that a graph answers "that isn't here" better than grep — did not reproduce; all four arms abstained correctly on every trap.

Two questions tip toward the graph arms by one cell each. That is noise, and is reported as noise rather than as a counter-finding — a discipline this page adopted after one of them was once misread the other way.

Reproducing it¶

node scripts/kgbench-run.mjs --set coding-v1 --preflight-only
node scripts/kgbench-report.mjs --run <run-id>

Stored answers mean a corrected grader can be re-applied offline without re-running anything.

Does a code-graph backend earn its keep, compared to an agent that just greps?

This repository maintains two code-graph backends — Graphify and CodeGraph — each of which costs an index, a container service, and a rebuild step. This benchmark exists to answer whether they buy anything a plain grep agent doesn't, using measurements rather than intuition.

Run: coding-v1-r9 · repo at d8a9b0647 · models claude-sonnet-5, rapid-proxy/claude-sonnet-5 · 16 questions × 4 arms × 3 reps across 3 agents = 384 cells, 0 contaminated, 0 tool escapes · continuation budget 1.

This page analyses r9. The generated files beside it do not. RESULTS.md and report.json in this directory were regenerated on 2026-08-19 for a later run — 16 questions × 1 rep, 128 cells, corpus 7924e45bd, continuation budget 2, merged from kgv1-telemetry and kgv1-opencode-fixed. That run is not comparable to this one: the report states that runs at different continuation budgets are not, and the corpus commit and rep count differ too. Read the two side by side only for their conditions, never for their deltas. r9's own generated report is preserved at .data/kgbench/runs/coding-v1-r9/report.json, which is what this page is checked against.

This run exists because the previous one was measuring a broken tool. r8 is kept throughout as the comparison, because the difference between the two is the clearest evidence on this page. Three defects found after r8 published are fixed here:

• CodeGraph's index now covers the tree under test. In r8 it indexed a different tree, so the arm was silently degraded to Read with a broken tool attached. This is the change that moves the most numbers on this page — see the bottom line.
• A correct abstention is no longer scored as a hallucination. The matcher bounded the gap between a negator and its head in characters; it is now bounded in words.
• The index containment check tests the artefact, not a mention of it, so a legitimate run at an older commit is no longer refused.

Read the arm comparison as a replication of x2, r6 and r8 on correctness — the headline null result has now held for four runs — and as a correction to r8 on cost, where one arm's figures turn out to have described a defect.

The generated tables live in RESULTS.md and are re-rendered from results.jsonl on demand. This page is hand-written around those numbers and is not reproducible from a re-render — see Reproduce it.

Bottom line¶

Read the arm comparison on claude only. It is the sole agent whose tool surface is actually enforced, so it is the only one where "the grep arm" means the agent could not reach a graph tool rather than merely wasn't configured with one.

Neither graph backend buys measurable correctness — every arm's median is 1.00 — but the cost of asking is no longer uniform, and that is new. Graphify runs 2.0× grep's tokens, 1.8× its latency and 2.1× its money. CodeGraph runs 1.05× grep's tokens and 0.93× its latency — it is now marginally faster than grep and costs about the same to run, while still costing 2.2× in dollars because its calls price differently.

That is a reversal of what this page said for four runs, and it is the clearest single consequence of the fixes. In r8 CodeGraph read 161,322 tokens and took 50.3s — 2.1× the tokens and 2.8× the latency of grep. Its index was serving the wrong tree, so the arm was quietly degraded to Read with a broken tool attached, and reading files is expensive. Given an index that covers the corpus, the same arm answers the same questions for grep's price. The old cost finding measured a defect, not a backend.

The hybrid arm is the one to read the others against, because it is the only one shaped like production: it has every tool and chooses freely. It is now the fastest arm on the page at 14.4s. Across 48 cells it made 227 tool calls, of which 20 were graph queries, all to CodeGraph — 18 of its 48 cells consult a graph, against 6 in r8. Given both indexes and no instruction either way, the agent still reaches for text search 91% of the time, but "overwhelmingly ignores the graph" is no longer the right description: it ignored a graph that did not work.

This is the one result on the page that a replicate makes stronger rather than weaker. Pooled over the four runs whose hybrid arm offers a byte-identical tool surface — r6, r7, x2, r8, 248 claude cells and 1,084 executed tool calls — 17 calls reach a graph tool: 1.57%, 95% CI [0.1%, 3.0%] once the interval is clustered on the question rather than computed over calls. No run is an outlier.

But a single rate is the wrong summary, and that is the sharper version of this finding: of the 16 questions, 4 ever elicit a graph call in any of the four runs and 12 never do in any of them. The agent is not sampling a strategy at a low rate — it applies a near-deterministic policy that happens to select the graph for a small, stable set of questions. See what the agent picks and, for why the interval moved, tool-selection.md.

The hypothesis the set was designed to test — that a graph index answers "that isn't here" better than grep — did not reproduce. All four arms abstained correctly on every trap.

Read this as a null result on 16 questions, not as proof that code graphs are worthless. One per-question difference runs against the graph arms and two now run in their favour.

A1 is the finding that survives, and it is the only one left against the graph arms. CodeGraph misses all three of its A1 cells (median 0.65); Graphify and hybrid miss one of three; grep misses none. Its answer is a six-line YAML comment, and neither backend indexes YAML — a fact about what a code index contains, not about how well it searches. It survived the index repair untouched, which is what makes it trustworthy: the fix that erased L2 left A1 exactly where it was.

L2 is withdrawn. All four arms now score 1.00. For four runs this page reported CodeGraph failing every one of its L2 cells and called it "the finding — nothing else is that clean". It was an artefact of an index that never covered the tree under test. With the index repaired the arm takes all of them, and its median tool calls on L2 drop from 12 to 2: it had been flailing against a tool that returned nothing. The cleanest result on the page was the cleanest measurement of a defect.

And two questions tip the other way — by one cell each, which is not a finding and is reported as such. On B3 and on T1 (an abstain probe) grep drops a single cell of three while all three graph-bearing arms drop none. Every median involved is 1.00, so this is invisible in the headline table and would be invisible in any per-question median: it shows up only by counting cells, which is the discipline this page adopted after B3 was once misread the other way. Two single cells are noise. What is worth recording is only that the sign has changed — for four runs no graph arm was ahead of grep anywhere, and the reason was partly that one of them was broken.

CodeGraph also collapses on A4, missing all three cells for a median of 0.33 where every other arm lands on 0.82 or 1.00. That reproduces the same collapse the r5/r6 re-measurements found at two older corpus commits, so it is now seen in three independent trees. A4 asks what pins CodeGraph's own environment variables and what owns index freshness.

The L2 figure was an artifact, and this run settles it. The 0.22 this page reported for four runs pooled two different things: cells that queried the index and scored 0.50, and cells in which the index was not reachable at all. Repaired, CodeGraph scores 1.00 on L2 in a full run at HEAD — not just in the targeted re-measurement that first showed it. A1, whose answer lives outside any code index, does NOT move — so the corpus-scope finding below stands on its own. See the CodeGraph index does not cover the tree under test and the re-measurement.

Everything else this page used to list here was a single run's noise. See which per-question results replicate.

The agent axis is large, and its ordering is not stable¶

Every figure is a median over that combination's ranked cells — the same denominator in every column. An earlier version of this table quoted opencode over a subset and its latency over everything; see reliability for why the subset existed and why it was wrong.

Every agent that produces an answer is equally correct: median 1.00 on every arm, every agent, without exception. What separates them is what they spend getting there. On the identical grep arm, opencode costs 1.71× claude's content tokens and copilot 1.11×.

This table read r8 until 2026-08-20, and the correction reverses which agent is expensive. In r8 copilot cost 1.85× claude and opencode 1.38×; in r9 those are 1.11× and 1.71×. The two agents swapped places. Nothing about the page said which run the table came from, and the numbers were plausible in both — see defect 44.

So the honest form of this finding is narrower than the heading it used to carry. The agent axis does not beat the arm axis here: across claude's four arms the spread is 2.02× (grep 69,164 to graphify 139,810), against 1.71× across the three agents on grep. What is true is that the agent axis beats every arm choice except being forced onto graphify — grep, codegraph and hybrid sit within 1.14× of each other, well inside the agent spread. And the agent ordering did not replicate across two runs, while the arm ordering did. Choosing the agent matters as much as choosing the retrieval strategy, and it is the axis this repository does not measure when it spends infrastructure on the other one.

What the continuation budget changed¶

x2 reported opencode answering 6 of 48 cells and concluded it "fails to answer 88% of the time". That number was an artifact of the harness and it is now withdrawn.

The three agents were never getting one turn each. claude's -p runs an unbounded agentic loop; copilot is launched with --max-autopilot-continues 20; opencode's headless run is a single session that ends at the first assistant step with text and no tool call. On analysis-shaped questions that step is frequently the one where it has finished investigating and is about to write — 36 of x2's 84 opencode failures had a complete answer sitting in stdout that was never written to the file. Measuring an agent at a budget of 0 against competitors at 20 measures the harness.

Every answer-file agent now gets the same budget: one continuation, meaning one chance to resume its own session and finish. Same 48 cells, same arm, same model:

The latency column cannot be filled in for x2. 43 of its 48 grep/opencode cells were retried, and every one of them recorded only its last attempt's clock — the defect described under reliability. r8's figure is repaired and charges each cell for every attempt it made, so putting the two side by side would compare a corrected number against an understated one and read as a slowdown the budget did not cause. Completion is the comparison this table exists to make, and it is unaffected.

85% of cells (41 of 48) needed the extra turn. Across the whole run, 83 of 96 opencode cells used it and 0 of 96 copilot cells did — copilot's own 20-continue autopilot was already absorbing the same failure mode invisibly, which is exactly why the asymmetry was hard to see.

It is not a retry. A retry re-runs the question from scratch, and for a deterministic narration-stop that just narrates again — x2 issued 88 retries and got 88 further no-results. A continuation resumes the session where the work has already happened.

What it costs. An earlier version of this section reported that the budget buys completion at the price of quality — mean score over answered cells falling from 0.977 to 0.948 while completion rose from 44/48 to 48/48. That trade-off does not replicate, and the number it was measured against was wrong.

The budget-2 run was repeated at the same settings on a corrected harness (coding-v1-r8-cont2b). On the same 48-cell comparison:

The claimed quality cost was −0.029. Between the two budget-2 runs — identical arm, agent, model, budget and questions — single questions move the 48-cell mean by −0.011, +0.021 and +0.018, which is the same size. The "cost" was one or two questions' ordinary run-to-run variance, published as an effect. Completion, which moves 44 → 48 in both runs, is real.

The shared-denominator figure was also arithmetically impossible as published (0.935). Budget 1's 44 answered cells sum to 43.00, so the mean over 48 is 0.896; reaching 0.935 would require those 44 cells to average 1.020, above the maximum score. Correctly stated, the budget's gain on a shared denominator is 0.896 → 0.975, which is larger than the retracted claim, not smaller.

The corollary matters for reading x2: its opencode median of 1.00 was survivorship. It was computed over the 13% of cells that happened to write, which were the easy ones.

This run was measured at budget 1. The repository default is 2, on evidence collected after it. At budget 1, 41 of 48 cells spent the entire budget and 4 failed to answer — the shape of a binding constraint. At budget 2 the spread over 0/1/2 continuations is 7/31/10 (cont2b; cont2 gave 9/28/11), and all 48 answer.

Note that about a fifth of cells still spend the budget in full — 10 of 48 here, 11 of 48 in cont2. An earlier version of this paragraph said "nothing reaches the ceiling", which contradicted the spread quoted in the same sentence. What actually changes is the consequence of reaching it: every cell that spent both continuations still answered, and answered correctly (10/10 and 11/11, median 1.00). The budget stops binding on the outcome rather than stopping being spent.

r8 is therefore not the run that demonstrates the current default, and runs at different budgets are not comparable to each other.

How to read this — the setup in plain terms¶

An "arm" is one way of answering¶

Every arm is the same model, the same prompt, the same questions. The only thing that differs is which tools it may use. That isolates retrieval strategy as the single variable.

The first three arms are forced onto one strategy, which is not how an agent works. That is deliberate — forcing is what isolates the variable — but it means none of them answers the question a maintainer actually has, which is "if I install this, am I better off?" Only hybrid answers that, and every other arm should be read against it.

Only claude can be held to an arm¶

--allowedTools, --disallowedTools and --strict-mcp-config are claude flags. For copilot and opencode the harness restricts an arm's MCP servers by writing the config file each CLI reads, but their built-in file and search tools cannot be withheld.

So on those agents an arm name describes the strategy a cell was asked to use, not one it was confined to. The two arms whose identity depends on withholding built-in search — graphify and codegraph, which grant Read but not Glob/Grep — are refused outright on copilot and opencode rather than run under a label they would not honour. That is why the agent table above has only grep and hybrid rows.

A "cell" is one complete agent session¶

One cell = one headless agent run: it reasons, calls tools, reads results, and writes a final answer. 16 questions × 4 arms × 3 reps on claude (192) plus 16 × 2 arms × 3 reps on each of copilot and opencode (192) = 384 cells.

Cells run strictly sequentially. Running them in parallel would be ~4× faster but would corrupt the latency and token measurements through CPU contention — and that is not hypothetical, see When the machine lied.

Where everything runs¶

graph TD
    R["kgbench-run.mjs<br/>(host) — spawns, scores, records"] --> C["one agent process per cell<br/>claude -p · copilot · opencode"]
    R --> S["sandboxed git worktree in /tmp<br/>= the arm's working directory"]
    C -->|"all model calls"| P["LLM proxy :12435<br/>(measured, subscription provider)"]
    C -->|"MCP over HTTP"| G["graphify server :3851"]
    C -->|"MCP over stdio"| K["codegraph"]
    G --> I[("indexes in .data/<br/>graph.json + codegraph.db")]
    K --> I
    subgraph container["coding-services container"]
        G
        K
        I
    end

The arm runs on the host, inside a throwaway copy of the repository. Its tools reach into the container. Grading happens back on the host after the answer is written.

How the answer is collected, and why it differs by agent¶

claude streams its answer as structured JSON. copilot and opencode are told to write the answer to a file, because an analysis-shaped prompt makes copilot exit within seconds and opencode yield at its first toolless step — both "succeeding" having answered nothing.

That difference is a confound in every cross-agent comparison here and it is not removable: it is what makes those cells produce an answer at all. It is also where this run's worst defect lived — see defect 15.

The five question classes¶

The abstain class is the interesting one, and the reason this set exists. Its questions ask about things that were genuinely removed from this repo, or never existed. A stale index answers them confidently and wrongly; grep comes up empty. That asymmetry is the most decision-relevant thing a retrieval benchmark can surface, and correctness-only scoring hides it completely.

The questions¶

All sixteen, verbatim — this is the whole test. "Correct" means the listed facts appear in the answer, checked mechanically rather than by impression.

lookup — one fact, one place¶

L2's key was corrected after the previous run. It had required lib/experiments/compare.mjs as the sole implementer. summaryStats is defined twice and independently, and the retrieval benchmark's copy is lib/kgbench/report.mjs — the question says "for the retrieval benchmark", so the key had named the other harness. Every arm had been answering correctly and scoring 0.15. r6 has since been regraded against the corrected key, which shows exactly that: grep 0.15 → 1.00, hybrid 0.15 → 1.00, graphify 0.15 → 0.83, codegraph 0.15 → 0.50. The claim above used to be an assertion about what the answers said; it is now a measurement.

structural — how pieces relate¶

blast — consequences of a change¶

arch — narrative, not location¶

A3 and A4 are replacements. The originals asked why an API key is stripped and how stdio differs from HTTP on crash. Both were answerable from general knowledge, and in an earlier pass 70 of 160 architecture cells answered them with no tool call at all. They measured the model, not the repository.

abstain — the answer is not in this repository¶

Retired: T2¶

What Cypher query does runCypherQuery execute to find callers of a symbol?

Written as an abstain probe on the assumption the Cypher path was gone. It is not. runCypherQuery still builds literal Cypher at integrations/semantic-analysis/src/services/cgr-query-cache.ts:233. Arms that produced the query were right and scored 0; the arm that abstained scored 1.

Retired for a false premise, not for scoring badly — dropping questions because results look wrong is selection. Its rows are excluded from every number here.

How answers are scored¶

score = required facts found ÷ required facts, checked against a per-question checklist of paths, symbols, and patterns. Any forbidden fact forces 0 and flags a hallucination — a confidently wrong answer is worse than an incomplete one, because on the receiving end that is the incident.

Every piece of ground truth is a file:line reference, machine-checked by scripts/kgbench-verify-questions.mjs, so a rename can't silently rot the answer key.

Why the arms can't cheat¶

The questions live in the repository the arms are asked to search. During piloting, the grep arm answered a trap question by reading the answer key and scoring 1.00. A leaked answer key produces correct answers, so it is invisible in the scores.

Arms therefore run against a sandboxed git worktree with 26 paths removed: the answer key, telemetry exports, session logs, agent instruction files, this published report, and the grading and containment modules themselves. Containment is then verified by grepping the tree for each question's own prompt, and the run aborts if anything survives.

graders.mjs and sandbox.mjs describe what a right answer looks like and which subjects are traps, so an explanatory comment in either is a crib — and that happened four times, three of them in comments written to explain the previous leak. Neither file is any question's ground truth, so removing them costs nothing and ends the category. Prose discipline had already failed; structure is what holds.

A file-level exclusion is not sufficient on its own. Graphify indexes markdown headings as graph nodes, so a document stripped from the tree can still reach the graph arms through the index — .graphifyignore has to match.

Results¶

Correctness: a tie everywhere¶

Each group of four bars is one question class; the four bars within it are the four arms, always in the same order. The figures are scoped to claude, for the reason given above: a bar pooling three agents with different enforcement has no meaningful midpoint.

A class median hides a bad cell, and here it hides four. lookup reads 1.00 for every arm while codegraph sits at 0.00 on L2 and graphify at 0.65; blast reads 1.00 while codegraph sits at 0.50 on B3; arch reads 1.00 while codegraph sits at 0.65 on A1 and graphify and hybrid at 0.82 on A4. Three or four questions per class means a weak question vanishes into the median. Always read the per-question table.

But read that table against a replicate, not on its own. Of those four cells, one replicates across runs — see which per-question results replicate. Within r8, grep does not score below 1.00 on any question; across r7 and x2 it scores 0.82 on A4, so that is a fact about this run rather than a property of the arm.

A winner is declared only at a ≥1.25× median gap with non-overlapping interquartile range. Anything weaker prints "tie", because at these sample sizes a 1.3× gap is not a result — it's a coin landing the same way three times.

Cost: not a tie¶

Content tokens are total tokens minus each arm's measured empty-run baseline. That matters: a large fixed floor of system prompt and tool schemas is charged on every call regardless of strategy, and it compresses every ratio. Content tokens are what actually separate retrieval strategies.

Graph queries pull substantially larger payloads into context than a targeted grep does. That is the core cost finding, and it runs opposite to the usual intuition that an index should be the cheaper path.

Which backend is most expensive swapped between runs — Graphify led in x2 at 2.4× grep, CodeGraph leads here at 2.1× tokens and 2.8× latency — so treat the ordering between the two graph arms as unresolved at n=48. What replicates is the direction: both are well above grep, and hybrid sits within 10% of grep on every measure.

The hybrid bar is the one that matters: give the agent everything and it costs what grep costs. The graph arms' extra tokens are not the price of having an index — they are the price of being forced to use one.

What the agent picks when nothing is withheld¶

hybrid had all ten tools: Glob, Grep, Read, six Graphify queries, and CodeGraph's explore. Over 48 claude cells:

Six graph calls out of 230 — 2.6% — and only 6 of 48 cells touched a graph tool at all. This is the single most decision-relevant number here: the infrastructure is available, free at the point of use, and declined.

Pooled across four runs¶

Every other per-question claim on this page shrank when checked against a replicate. This one did not, so it is worth stating at full strength. The hybrid arm's tool list is byte-identical in r6, r7, x2 and r8, which makes those four runs poolable:

1.57% of tool calls, 95% CI [0.1%, 3.0%]. Under a single rate of 1.57% the expected counts are 5.0 / 4.8 / 3.5 / 3.6 against observed 3 / 5 / 3 / 6 — every run within Poisson noise, no outlier (smallest tail p = 0.16, r8). Four runs, four trees, two answer keys, and the agent's appetite for the index does not move.

That interval is the cluster-robust one, and it is 4.0× wider than what this section published for four runs. The figure used to read 95% CI [0.8%, 2.3%], which is the naive binomial interval over 1,084 calls — and it treats those calls as 1,084 independent draws. They are not. They come from 16 questions asked three times each, and the reps of one question agree with each other almost perfectly: in every run, 14 to 16 of the 16 questions have all three reps making the same choice. Clustering on the question — the unit that actually varies — the interval is [0.1%, 3.0%]. The point estimate does not move; the precision claimed for it was never earned. The same correction applies to the per-cell figure, which is why it is no longer quoted.

A better statement of the same data: of the 16 questions, 4 ever elicit a graph call in any of the four runs, and 12 never do in any of them. The four are A1 (1 run of 4), A4 (3), B1 (3) and B2 (1). "1.57% of calls" averages a bimodal population and describes no member of it — the agent is not sampling a strategy at a low rate, it is applying a near-deterministic policy that happens to select the graph for a small, stable set of questions. See tool-selection.md.

Graphify specifically accounts for 3 of the 17 and is untouched entirely in two of the four runs. Its count moving 0 → 2 → 0 → 1 between runs is not a trend; at these numbers each step is one or two cells.

An earlier version of this section cited r6 as "4 graph calls in 348". Both figures were wrong — it is 3 in 322 — and no run, arm or agent in the corpus produces 4/348. Recomputed here from results.jsonl, and the claims checker now derives them rather than matching text.

Two honest caveats. The tool descriptions are what the agent chooses from, so this measures the appeal of the advertised interface as much as the index behind it — a better-described graph tool might get picked more. And a preference is not a justification: the agent could be choosing wrong. But it is not choosing wrong and paying for it, because its correctness matches the forced-grep arm exactly.

This finding has since been investigated in its own right, and three of the claims in this section need qualifying. See tool-selection.md. In short: the pooled figure is claude-only (copilot and opencode record no tool calls at all, so the obvious cross-agent test cannot currently be run); the cell-level statistics here are too precise, because 48 cells are 3 reps of 16 questions and reps are near-perfectly correlated; and the choice is not a rate but a near-deterministic function of the question, committed on the first tool call and essentially never revised. The tool-description caveat above turns out to be causal, and it has now been tested directly. Restating Graphify's six tool descriptions in applicability-first form — same names, same schemas, same handlers, same results, same pinned corpus — moved it from 0 calls across all 48 cells to 11, and from 0 of 16 questions to 6 (McNemar exact p = 0.031). It acts on the opening tool call, and it cost nothing in correctness. Most of what it won came out of the other graph backend rather than out of grep, so total graph share moved only 8.8% → 12.0%.

The forced arms show a milder version of the same reluctance: given only their backend, the graph arms still answered without making a single graph call in 6 of 48 cells each.

Where corpus scope shows up¶

Every point lost in this run belongs to a graph arm: grep scores 1.00 on all sixteen questions here, and the graph arms lose on four between them. Only one of those four survives a replicate — see below before quoting any of them.

Every dot is one architecture-class cell, so it shows A1 and A4 but not L2 or B3, which belong to other classes and appear in the table below rather than the figure.

Four of the five classes tie at 1.00 across all four arms. arch no longer does: CodeGraph's median there is 0.65. That is new, and it is the first class-level difference this benchmark has produced. It is also entirely two questions — A1, whose answer is a YAML comment no code index covers, and A4, where CodeGraph misses all three cells. A class median that moves because both of its hard questions moved is a restatement of those two questions, not an independent finding, and it is quoted here as such. The other three arms remain at 1.00 in every class.

L2 is 0.00 on all three codegraph reps, missing both required facts, and it is not a case of not trying: codegraph spent a median of 12 tool calls there against grep's 7, and still missed. That is the one per-question result that has now replicated — L2 was 0.00 for codegraph in x2 as well.

The mechanism is corpus scope. CodeGraph indexes code entities into SQLite/FTS5. When an answer lives in prose inside a config file — a _comment or _envNote key — comment strings are not code entities, so the index cannot reach them. Rather than find nothing and say so, the arm answers from the one source it does have, its own MCP tool description, and produces a fluent, confident account of constraints that are not this repository's.

That is worth stating plainly: the failure mode of a too-narrow index is not an empty result, it is a confident answer from whatever else is in context. Grep, which has no index at all and just reads the file, scores 1.00 on all four.

The same mechanism, on the other backend¶

A1 makes the point a second time, for Graphify, and the file boundary is sharper. The question asks why .observations is not bind-mounted. The entire answer — all three checklist facts — exists in one place: a six-line YAML comment at docker/docker-compose.yml:82-86.

Graphify indexes no YAML at all. Of the 4,346 distinct source files in graph.json, zero are .yml or .yaml, so that comment is not narrowly missed, it is categorically outside the corpus. The consequence shows in what the arm cites: across 16 A1 cells Graphify names docker-compose.yml twice, against grep's 14 of 16.

All five Graphify failures miss the identical fact — f1, the SQLite WAL/SHM corruption rationale — and none ever miss f2 (obs-api is the single owner) or f3 (the container reaches it over HTTP). That asymmetry is the mechanism in miniature. obs-api appears in 284 files and OBS_API_URL in 22, so both have a structural footprint the graph can reach. The reason has none: it is prose, in a comment, in a file type the indexer skips.

And again the arm does not abstain. Side by side, same question, same run:

grep (1.00, 3 tool calls) — "Previously the container and host both opened the same SQLite observations.db across the Docker bind-mount boundary, which caused WAL/SHM corruption from concurrent access."

graphify (0.65, 5 tool calls) — "This confirms the mechanism and file evidence. I have enough to answer directly. […] the obs-api HTTP service, which owns the LevelDB lock […] specifically to avoid two writers racing on the shared store."

The shape is right and the substrate is wrong: the store is SQLite, not LevelDB. LevelDB is in Graphify's index — it backs .data/knowledge-graph/ — so the arm reached for the nearest indexed neighbour and delivered it with a commit hash and a confident preamble.

One caveat, because this is one question. Failure is probabilistic rather than deterministic: the WAL/SHM fact also survives in docs/observations/README.md and system-health-dashboard/server.js, both of which Graphify does index, which is why 9 of its 14 non-citing cells still score 1.00.

CodeGraph fails A1 the same way, and the earlier verdict here was a measurement artifact¶

This page previously said CodeGraph's A1 losses were a different, unexplained problem: it "cites docker-compose.yml in 10 of 16 cells and still scores 1.00 in only 4 of those 10", so "reaching the file is not its problem on A1, and what is remains unknown." Both sentences were wrong, and the metric behind them was measuring its own opposite.

The citation test was /docker-compose/i.test(answer) — the file name appearing anywhere in the text. Six of those ten "citations" are the arm saying it could not read the file:

"no live access to this repo's docker-compose right now — file tree here is stripped down for the benchmark" — r8 rep1, 0.65

"I can't grep the actual docker-compose.yml […] (no Bash/Grep tool, and this working tree has no .codegraph index), so treat this as inference from adjacent memory" — r8 rep3, 0.65

"the real 'coding' repo isn't checked out here, so I can't verify docker-compose.yml directly" — r7 rep7, 0.65

The other four are the opposite claim, and they are the four that score 1.00: "Found the exact answer in docker/docker-compose.yml:82-86." A keyword test counted the denials as citations. Reaching the file is precisely CodeGraph's problem on A1.

Classified by what the answer claims rather than which words it contains, all 64 A1 cells across the four arms line up on one axis:

Twenty-nine cells claim to have read the file and twenty-nine score 1.00, with no exceptions in any arm. The middle row is CodeGraph alone.

The repository was not absent. The run's sandbox is a full worktree at f4f13e86a (run.json records mode: worktree, verified: true) whose exclusion list covers .data, the session logs (.specstory at the time; .coding since per-repo tenancy), CLAUDE.md and .claude — anti-leakage, not the source tree. docker-compose.yml is present and carries all three facts at lines 82-86, which is how grep reads it in 16 of 16 cells. CodeGraph is the only arm that ever asserts otherwise, and it does so far more widely than A1: at least 18 cells of 172, against 0 for grep, 0 for hybrid and 0 for Graphify. (An earlier version of this paragraph said 7, from the same phrase-enumerating regex that under-counted A1; 18 is a widened count and still a lower bound, since only A1's sixteen cells have been read individually.)

Denying access is not uniformly fatal — it is fatal when nothing else carries the answer:

Where the fact is reachable from the index or correctly held in memory, the arm declares the repository absent and answers correctly anyway. Where it is not — A1 and L2, the two questions this page already identifies as living outside the index — the same declaration precedes the worst scores on the page.

Why it happens is the same mechanism as Graphify's, one level up. The codegraph and graphify arms are defined as Read plus their backend's MCP tools and no Glob, no Grep — deliberately, since that is what distinguishes a code-graph arm from the baseline. But Read needs a path you already have. With no search tool, the index is the path-discovery mechanism, so a file the index does not cover is not merely un-summarised, it is unaddressable. CodeGraph indexes code entities into SQLite/FTS5 and a YAML comment is not a code entity, so the arm gets back no path — and from that it infers, wrongly, that the file does not exist. Four times in sixteen it guessed docker/docker-compose.yml from prior knowledge and scored 1.00; twice it recovered f1 from indexed prose instead (scripts/observations-api-server.mjs, docs-content/core-systems/observational-memory.md), the same redundancy that rescues 9 of Graphify's 14 non-citing cells.

So A1 is not two mechanisms. It is one: the answer lives in a file neither index covers, and neither arm has a tool that can find a file its index does not already know about. The two backends differ only in bedside manner. Graphify asserts LevelDB with a confident preamble; CodeGraph discloses and then asserts LevelDB anyway. Every one of the 10 failures names LevelDB and none names SQLite. The disclosure is worth something and it is not worth much: the score is 0.65 either way.

Every CodeGraph A1 cell was then read end to end, because the classification above is regex-derived and a regex is what produced the retracted verdict in the first place. Hand-audited, all sixteen partition without a residue:

Every cell that claims to have read a file scores 1.00 (6 of 6); every cell that claims no file fails (10 of 10). No cell reads a file and misreads it, and no cell answers from memory and gets it right. That is a cleaner partition than the pooled table above, and it is only visible by reading.

It also corrects a number this page published one commit ago. The claim was that "five failing cells make no claim about the file in either direction". It is two. The regex behind the five had enumerated the phrasings it expected — it missed r7 rep8's "no .codegraph/ index or file access available in this sandbox", which is a denial in words the pattern did not list, and it counted r7 rep1 and rep10 as silent when both name memory as their source. Of the ten failures, eight disclose that they are not reading from the repository. Only two are genuinely silent — r7 rep4 and rep6, which state the LevelDB account flat, with no hedge and no source, indistinguishable in the text from an arm that looked and misread.

The CodeGraph index does not cover the tree under test¶

Chasing the L2 zeros turned up a defect in the harness rather than in the backend. The five CodeGraph L2 cells that score 0.00 are not wrong answers. They are refusals, and they are correct:

"I don't have a way to search this codebase right now — there's no .codegraph index for this project, and I don't have Grep/Glob/Bash tools available in this session (only Read and codegraph_explore). I can't reliably locate summaryStats or its importer without guessing at paths, which I'd rather not do." — r8 rep2, scored 0.00

Three of the five ask the operator to run codegraph init. That is verbatim what the tool tells them, and the reason is structural:

The CodeGraph MCP server is a container-side stdio process (docker exec -i coding-services codegraph serve --mcp). The sandbox worktree is invisible to it, and its default project resolves to an uninitialized directory, so a call succeeds only if the agent supplies projectPath: /workspace/coding — a container path it cannot derive from its own environment. Reproduced directly against the running container:

$ docker exec -w /workspace/coding coding-services codegraph status
  Project: /workspace/coding          Index Statistics: …
$ docker exec coding-services codegraph status          # default cwd /coding
  ⚠ Not initialized    ℹ Run "codegraph init" to initialize

Two consequences, and the second is worse than the first.

It sometimes returns nothing. At least 30 of 172 CodeGraph cells report the index unreachable (a lower bound — the count comes from the answers' own words). It is not uniformly costly: on T4 all nine such cells still score 1.00, because that question does not need retrieval. It is very costly where the question does.

When it does answer, it describes the wrong tree. A successful codegraph_explore serves /workspace/coding — the main repository — while every other arm searches the de-contaminated worktree. The sandbox verifies containment by grepping the tree it builds; that guarantee does not extend to an index of a different tree. 15 of the 31 sandbox-excluded paths are present in the index, including lib/kgbench/graders.mjs, judge.mjs, sandbox.mjs and ten test files. The answer key is not among them (config/kgbench/questions/ is JSON, and no JSON is indexed), nor are the observation exports (.data/ is not indexed) — so the contamination the sandbox was built to stop did not get through this hole. No cell demonstrably exploited it either: six answers across all four arms name an excluded path, but grep and hybrid do so without any index, so those are inferences, not retrievals.

The index is also pinned to its own build commit rather than the run's — 6ecbbe7f against f4f13e86a. Here that is one file apart and immaterial, but nothing enforces it.

What this costs the L2 result. Split by whether the index answered:

The published 0.22 blends a harness defect with a capability result. The capability result survives and is cleaner than the blend: given a working index, CodeGraph named lib/kgbench/report.mjs correctly and scripts/kgbench-charts.mjs wrongly in four cells out of four, where grep and hybrid score 1.00. The 0.22 is not a number about CodeGraph.

The scoring is what hid it. A cell that says "I cannot search this repository, and I will not guess" scores 0.00, while a cell that names one right file and one wrong one scores 0.50. On this question the rubric pays better for guessing than for an accurate report of a broken tool — which is exactly backwards from what you want an agent to do, and it is why five correct diagnoses sat in the data for three runs looking like a retrieval failure.

The index was fixed and the two affected arms re-measured (coding-v1-r8-cgidx)¶

The harness was repaired and codegraph and hybrid re-run at the same corpus commit (f4f13e86a, pinned with --commit), same questions, reps, agent, model, judge and continuation budget. The index is the only variable. 96 cells, all ok, run id coding-v1-r8-cgidx; provenance in that run's REINDEXED.md.

The categorical result, which is the one this run is entitled to claim: cells reporting an unreachable index went from 14 to 0.

L2 was the page's cleanest per-question finding, and it was an artifact. CodeGraph took all nine of its cells once the index covered the corpus. What the arm had been failing was not the question but the tool.

A1 did not move, and that is the more informative half. Still 0.65, still missing exactly f1. Its answer is a six-line YAML comment and CodeGraph indexes no YAML, so the corpus-scope finding above stands on its own and is independent of the harness defect. Had A1 moved, the explanation this page gives for it would have been wrong.

A4's drop is not attributed to the fix. Its own history across runs is 0.82 (r7), 0.49 (x2), 0.94 (r8); 0.54 sits inside that spread, on a question this page already classes as bimodal with no arm effect. Three cells within a question's known range is not an effect, by this page's own rule.

Hybrid's T3 first read 0.67 here, and that was a grader defect rather than an arm effect; it is now fixed and regraded. Not one of hybrid's sixteen questions moved in either direction.

The change is behavioural, and it is larger than any score¶

CodeGraph stopped compensating by reading files and started using its index. That reframes every earlier CodeGraph number here: they measured an arm quietly degraded to Read, with a broken tool attached.

This qualifies the central result of this page. "Agents overwhelmingly choose grep" was measured while one of the two graph backends returned nothing. With it working, hybrid reaches for a graph tool in 20 of 48 cells instead of 6. The direction survives — grep and Read still account for 89% of its calls, and no question's score improved — but the honest statement is narrower than the original: agents preferred grep, and preferred it far more strongly when one backend was broken; with it fixed they consult the graph three times as often and answer no better. A tool being reached for and a tool being useful are different claims, and only the second one is still clean.

Hybrid's T3 was a grading defect, since fixed and regraded¶

T3 is an abstain probe. One cell scored 0.00 with hallucinated: true, which would have been the worst kind of regression — except the answer abstains, twice:

"There's no "payment reconciliation" module in this repo …" "No payment-processing/financial-transaction reconciliation module exists in this codebase."

Both were missed by the matcher. The first because \bno (?:module|…)\b required the noun immediately after no, and a quoted subject sat between them. The second because \bno\b[^.!?]{0,60}\b(?:exists?|…)\b allowed sixty characters between No and exists and this subject is sixty-four. The arm answered correctly and lost the cell by four characters.

Both bounds were the same bound — how much may sit between the negator and its head — and both were counted in the wrong unit. The gap is now measured in words, not characters, because in a code repository the subject of an abstention is a hyphenated, slashed, dotted compound: a character budget tightens exactly as the subject gets more technical, which is backwards. Three words, sixty-four characters.

The fix moves three cells in three runs and nothing else — r8 grep/T4, x2 grep/T3, and this run's hybrid/T3, each 0.00 → 1.00 with hallucinated cleared. All three were regraded from the stored answers, so no cell was re-run and no measurement changed; the pre-regrade files sit beside each results.jsonl. Hybrid's arm mean is 0.985 both before and after the index fix. The apparent regression was entirely the grader's.

That the failing cell was also the only one of the three to call a graph tool is not coincidence: the graph surfaced the token-reconciliation files, the arm named them while ruling them out, and the longer subject phrase overran the window. A working tool made the answer more informative and the grader punished it for the extra words.

Which per-question results replicate¶

A three-rep median is a fragile statistic, so every per-question claim above was re-checked against the two other runs that share this answer key (r7, x2).

Counted in cells, not medians. A per-question median hides a bad cell exactly the way a class median hides a bad question, and it did: reading medians alone, B3 looks like an r8 artifact, because its x2 failure is the minority cell in a 1.00 / 1.00 / 0.00 triple. The table below therefore counts every claude cell scoring below 1.00, pooled over the three runs:

L2 is the finding. CodeGraph scores below 1.00 on every one of its nine cells across three runs, and never exceeds 0.50. Nothing else on this page is that clean.

A1 is the second finding, and it implicates both backends. CodeGraph misses 10 of 16 across two runs and Graphify 5 of 16 across two runs, against 0/16 for grep and hybrid. Earlier versions of this page named CodeGraph on A1 and not Graphify; the cell counts say both.

B3 was previously labelled an r8 artifact here. That was wrong — CodeGraph also fails a cell in x2, which its 1.00 median concealed. Three failing cells across two runs is real but thin, and it is quoted as such rather than as a finding.

A4 is not a finding at all, and this page previously reported it as one. It is a two-value question — every cell scores 0.82 or 1.00 — so a three-rep median is decided by which value happens to land twice. In r8 all four arms produced both values, and the medians split 2–2 by coin flip. Pooled over 16 cells per arm the spread is 0.78–0.89 with hybrid the highest, and r7 alone, at ten reps per arm, puts all four arms at exactly 0.82. The sentence claiming graphify and hybrid drop A4 is withdrawn.

grep is not 1.00 everywhere either, once you stop reading medians. It is 1.00 on all sixteen questions in r8 — a fact about this run, which earlier versions of this page generalised. Pooled over the three runs it drops cells on four questions: A4 (10 of 16), and one cell each on B1, B3 and T4. The T4 cell is its one hallucination. It was five questions and two hallucinations until the abstain matcher was fixed: its x2/T3 cell was a correct abstention the grader could not parse.

What survives across three runs is narrower than the old wording and still worth having:

• No graph arm beats grep on any question in any run. The direction never reverses — except once, on three cells, in a run added later. See the re-measurement below.
• L2 fails the same way every time, and A1 fails for both backends across two runs.
• Which other question a graph arm drops is noise, and A4 is not an arm effect at all.

r5/r6's rewritten questions, re-measured beside them¶

A3, A4 and B1 were rewritten after r5 and r6 ran, so those runs' scores for them describe a question that no longer exists and cannot be regraded (defect 41). They have now been re-measured rather than patched: two new runs, coding-v1-r5-requestions (69 cells) and coding-v1-r6-requestions (52 cells), each pinned to its own run's corpus commit (199bf1f3f, fcfedffaa) with that run's arms and rep counts, on claude-sonnet-5.

They are deliberately NOT spliced into r5/r6. Those runs recorded no model at all — not run-level, not row-level — and judged with claude-opus-4.8. Overwriting three questions inside them would leave one model answering A3/A4/B1 and an unknown one answering the other thirteen, confounding arm with model on exactly the questions under test. A run's whole purpose here is a within-run arm comparison, so the repair would have destroyed the thing being repaired. Beside them, they cost nothing and confound nothing.

121 cells, all ok, zero hallucinated, zero contaminated.

Read the arm columns, not any before/after: the old scores answer different words, so a delta across them mixes question, model and date. Within each re-measurement the question, model and corpus are constant, which is what makes arm-vs-arm clean.

The one result that is strong: CodeGraph collapses on A4, in both runs. 0.61 and 0.56 against 0.87-0.94 for every other arm — the largest arm gap anywhere on this page, replicated across two independent corpus commits older than any run behind the tables above. A4 asks what pins CodeGraph's own env vars and what owns index freshness; the arm that is CodeGraph is the one that cannot answer it.

And the one that costs this page a claim. On r6's B1, graphify and codegraph both score 1.00 against grep's 0.94 — the first time in this corpus that a forced-graph arm finishes above grep by more than the 0.03 floor. By this page's own standard it is far too thin to be a finding: grep lost exactly one cell of three (rep3 = 0.82), and at three reps one cell is 0.06. It is quoted for the same reason B3 is — not because it shows anything, but because "the direction never reverses" was an absolute, and an absolute is refuted by one counter-example or it is not an absolute. The honest version is the direction reverses only where a single cell can reverse it.

Everything else replicates the published direction. A3 is a three-way tie at 1.00 — no arm effect at all. A4's spread flips grep between first (r5, 0.93) and third (r6, 0.87), which is exactly the two-value behaviour this page already describes for A4, on the same question text.

Reliability¶

378 of 384 cells answered. No stalls, no timeouts, no tool escapes, no contamination. All six failures are opencode, and all six are the same termination behaviour that budget 1 does not fully cover: the agent finished investigating, said so, and stopped without writing. Their stdout tails read "I have enough detail now" and "I have enough. Let me write the answer." Two separate 48-cell runs at budget 2 (coding-v1-r8-cont2, and coding-v1-r8-cont2b on the corrected harness) each answered all 48.

Latency tails: p90 is 34.9s for grep, 32.4s for hybrid, 69.2s for graphify and 122.0s for codegraph. The arm with every tool available has the tightest tail of all — the same inversion x2 found.

Three hallucinations in 384 cells, and all three are the arms with text search. Two are T4 and one is T1 — abstain questions, where the correct answer is that the thing does not exist. grep/claude fabricated once, grep/copilot once, hybrid/opencode once. Every cell scored 0.00 for it.

This count was four until the abstain matcher was fixed; the fourth was a correct abstention the grader could not parse. Two of the five hallucinations this page once reported across all runs were the grader's, not the arms'.

Neither forced graph arm hallucinated. An earlier version of this page called that "the one result that favours an index", hedged it as too small to lean on, and leaned on it anyway. Checked against the other runs, it is indistinguishable from chance and is withdrawn.

The per-run counts are 0, 0, 0, 3 for r6, r7, x2, r8. This run is the high outlier, not the typical case — and it is now the only run with any hallucination at all. The cleanest comparison is claude alone — the only agent that runs all four arms, so arm and agent are not confounded — which gives a perfectly balanced 72 abstain cells per family across the four runs:

At the pooled 0.7% rate the chance of seeing zero in the graph arms is P = 0.61. Pooling all three agents gives 3/144 against 0/72 and P = 0.37. Neither is remotely near significance — and both moved further from it when the grader was fixed, which is the direction a withdrawn claim should move. The framing also fails inside its own family: both claude hallucinations are grep's, and hybrid — which has text search too — has none.

Settling this would need roughly 400 abstain cells per family against the 72 available, a purpose-built run several times the size of this one. Until someone funds that, "graph arms don't fabricate" is not a result this benchmark has. What the four rows do support is narrower and still useful: abstain questions are where fabrication shows up at all — every hallucination in every run is T-class.

The token-attribution warning on this page was wrong, and it has been withdrawn. An earlier version said 21 of opencode's 96 cells double-counted a neighbouring cell's session, and excluded them from every opencode figure. Both the cause and the remedy were mistaken.

Those 21 cells are exactly the 21 cells that were retried. A retry is a fresh spawn, so it opens a session of its own; the resolver judged ambiguity per cell, saw two sessions, and flagged every retried cell in the run. The arithmetic that seemed to confirm the neighbour theory — one flagged cell's 274,139 tokens being "its own 139,727 plus its predecessor's 134,412" — was reading the same cell's first attempt as a predecessor. The predecessor cell was a third session, 172,223 tokens, never counted at all. Checked across the whole run, no session's start falls inside more than one cell's window: there was no bleed to find.

So the sums were right and only the label was wrong — and excluding those rows made the numbers worse, not better. A retried cell pays for two attempts, so dropping the retried cells dropped the expensive ones: it pushed opencode's measured cost down. Restoring them moves its content-token median from 90,109 to 107,170 on grep and 113,145 to 121,469 on hybrid, and its cost relative to claude on an identical arm from 1.16× to 1.38×. The correction makes opencode look worse, which is the direction that says the exclusion was not protecting anyone.

The underlying defect was in the runner, not the resolver. A cell's tokens were resolved over a window spanning every attempt, but the row was built from the last attempt — so it recorded that attempt's clock beside an all-attempts token total, and could not reproduce its own number. wall_s was understated by the same mechanism: this section's own latency figures were medians over cells charged for one attempt out of two. All of it is fixed; the run's rows were repaired in place from the proxy DB, without re-running a cell, and the repair is checked by requiring that per-attempt attribution and whole-span attribution agree to the token. No claude or copilot figure on this page moved.

What this does not show¶

• One repository, one question set. Nothing generalises to other repos without re-running.
• hybrid measures a preference, not a verdict. It shows what these models pick from these tool descriptions. A graph tool described differently could be picked more often. It does not show the graph is useless — it shows it is unchosen.
• Indexing cost is excluded. Per-query numbers ignore what it costs to build and keep the indexes fresh. That is a real expense on the graph side, so the graph backends look better here than their true total cost.
• The indexes match the tree in this run. Both were rebuilt at f4f13e86a immediately before launch, so unlike x2 — where the graph was 71 files behind what the arms searched — staleness is not available as an explanation for the graph arms' losses here. They lost against a current index.
• Corpus scope differs between backends (graphify indexes docs and PDFs; code-only backends do not), so node/edge counts are not comparable at face value.
• 16 questions is small, and arch is only 4. A null result at this size means "no effect detected", not "no effect exists".
• Cross-agent comparison is confounded by elicitation. claude streams structured JSON; copilot and opencode write to a file after a differently-shaped prompt. The confound is not removable — it is what makes those cells answer at all.
• Only claude's arms are enforced. copilot and opencode keep their built-in search on every arm, so their grep and hybrid rows differ by MCP configuration alone.
• 21 of opencode's 96 cells had their tokens re-resolved after the run, because the runner recorded a window that did not cover the attempts it made. This page and RESULTS.md now agree on every figure; an earlier version of this page quoted a subset and did not.
• Seven of those 21 rows gained tokens between the run and the repair — six by about 700, one by 26,661 — because the proxy's token DB is append-only and the stop-adapters write late. The repaired rows carry the later, higher numbers; the rows that needed no repair carry what they were resolved to during the run. Exact reproducibility of a token figure is only ever "as of when that row was resolved".
• This run was measured at continuation budget 1, and the repository default is now 2. A run at a different budget is not comparable to this one on either completion or cost.
• The scores are not comparable to x2's. x2 was graded by a mixture of haiku and opus; every judged cell here was graded by claude-sonnet-5. Re-grading x2 under this judge, not comparing the two tables, is the way to put them on one scale.

Where the disagreements went¶

Twenty cells out of 384, and every one of them checklist_higher — the deterministic checklist scored the answer above the judge, never once below. They fall on five questions:

This table is an alarm, not a diagnosis. It says two graders differ; it does not say which is wrong, and across every investigation on this set the cause has been a judge rubric, a false answer key, a regex, a shared match token, or a matcher too loose and too narrow at once — never a badly written question. Twice the arms were right and the key was wrong.

Read the by-arm row first. The disagreements are spread across all four arms — 6, 1, 6, 7 — and every single one points the same direction. A defect that moves every arm alike is a grader property, not an arm property: this is the judge applying a stricter reading of B2, B3, A1, A2 and A4 than the checklist does, on whoever answers them. It is a calibration gap between the two scorers, and the questions it concentrates on are the ones worth re-reading.

The detector is also blind to the most common defect of all: because the judge's prompt is built from the same checklist, a wrong key makes both graders agree and produces zero disagreements.

Unlike x2, one model graded everything. All 308 judged cells were scored by claude-sonnet-5 via copilot. One cell — hybrid/copilot S1 rep1 — was initially graded by claude-haiku-4-5 when a transient copilot failure dropped the judge onto the claude-code fallback, which ignores model selection; it was re-judged individually and returned the same score. The manifest keeps that substitution event on record rather than erasing it.

The 76 unjudged cells are not a judge failure: they are the abstain questions, which carry no checklist and are never judged by design, plus the six no_result rows.

Every median and ranking on this page uses the deterministic checklist score, so none of them depends on the judge at all.

Provenance of these numbers¶

All 384 cells come from a single uninterrupted pass at tree commit d8a9b0647 — no resume, no splice, no re-run. x2 needed a page of provenance because half of it was re-run after a defect; this run needs a paragraph.

One thing this run does NOT share with r8: copilot's 95 token rows were filled in afterwards by kgbench-backfill-tokens.mjs. That is routine rather than a repair — copilot's stop-adapter writes its row about a minute after the cell ends, long after the runner has recorded unmeasured, and waiting per-cell would cost hours of sleeping. Every cell is measured now (192 stream-json, 192 proxy-db-session, 0 unmeasured, 0 ambiguous), and the backfill joins on task_id and the stored wall-clock window rather than on proximity.

Three things about it are worth stating anyway, because each would otherwise be invisible:

• Both indexes were rebuilt at d8a9b0647 immediately before launch, so the arms and the graph backends saw the same tree. In x2 the index was 71 files behind. Staleness is therefore not available as an explanation for anything here.
• The working tree was dirty at launch (dirty: true). Arms search the commit, not the working tree, so this affects nothing they could read; it is recorded because a reader comparing to a clean-tree run is entitled to know.
• Two token figures were reconstructed after the run, not measured during it, and both are labelled as such in the manifest:

The floor is a property of the combination, not of any question, which is what makes measuring it afterwards legitimate. It is disclosed on every affected row as baseline_post_hoc rather than silently merged with the floors measured inline.

What went wrong building this¶

Forty-four defects were found across the runs behind this page, and runs were discarded repeatedly — two are still on disk carrying VOID in their name (coding-v1-VOID-tool-escape, coding-v1-x1-VOID-kb-injection), and a third, coding-v1-x2, was partially voided and repaired rather than thrown away. Every discard came from a defect that would have produced a plausible, publishable, wrong result. They are documented because the failure modes generalise to any agent benchmark.

| 19 | The harness answered its own question. A comment in runner.mjs reproduced L1's prompt verbatim and named install.sh — L1's answer. The file cannot be excluded from the tree: it is B2's and A2's ground truth, so it has to stay readable and be clean. | The leak scanner saw it and let it through. It derives five overlapping windows per prompt and needs three before a hit is decisive; a one-line quotation matches two, so it was filed weak and the run proceeded. Every arm that grepped L1's subject was handed the answer by the thing grading it. Leak #5 in a series where each was a comment explaining the previous leak. | | 20 | The fix leaked twice more. Replacing the subject left the sentence frame matching two of L1's windows — a crib is the question's form as much as its noun — and the paragraph documenting that put the frame straight back into the tree. | Three passes to remove one comment. The control is now mechanical instead of editorial: a needle hit anywhere under lib/kgbench/ or scripts/kgbench-* is decisive regardless of window count, because between the questions and the code that runs them there is no shared vocabulary to tolerate. Prose about what not to write is not a control. | | 21 | The answer-file directive read as a prompt injection. agents.mjs carries the instruction verbatim — write your complete answer to <file>… the task is complete ONLY once it exists — and it was inside the searchable tree. opencode found it, correctly classified it as an injection attempt, announced it was ignoring it, and stopped. | claude never sees the directive; copilot complied with it 96 times without comment. So the file penalised exactly one agent, which is the specific way a cross-agent comparison stops meaning anything. Now excluded outright; no question's evidence points there. | | 22 | One agent had no turns. claude's -p runs an unbounded loop, copilot is launched with --max-autopilot-continues 20, and opencode's headless run ends at the first toolless step — a budget of zero. | That asymmetry, not retrieval, is what x2's 88% opencode failure rate measured. 36 of its 84 failures had a finished answer in stdout with only the write missing. Every answer-file agent now gets the same budget, recorded per run and per cell. Retrying does not fix it: x2 issued 88 retries and got 88 further no-results, because a deterministic narration-stop just narrates again. | | 23 | A 6% failure rate appeared in the record once. opencode's CLI rejected 5.9% of its own bash calls for omitting a required description argument — 35 of 589. Exactly one reached a results row, because stderr is persisted as slice(-300) and only the occurrence that happened to land last survived. | Worse, a cell that answers keeps no stderr at all, so a rejection on a successful cell had no channel to the record whatsoever. A rate that shows up once reads as a curiosity, which is how it went uncosted through a 384-cell run. Now counted from the full buffer before truncation, and reported per agent as measurement provenance rather than as a score. | | 24 | A pin that was applied was then discarded. The judge was pinned to claude-code/claude-opus-5, and the proxy logs show the pin being honoured — then RATE_LIMITED, then a CLI worker-pool fallback that returns claude-haiku-4-5 whatever model it was asked for. The worker is spawned under key=claude-opus-5 and still answers as haiku. | On one day that was 21 opus calls against 2,065 haiku ones, and the haiku stretch covered all of x2. Availability was never the problem — the model is served fine when the direct path is up — reachability under load was. Probing establishes that a provider can serve a model, not that it will. The judge is now pinned to a provider that honours the model rather than the one with the best catalogue. | | 25 | A baseline that misses its window is gone for the whole run. grep/copilot's floor was measured with a single probe and a 150s wait; the probe's rows never arrived, and a cell's content_tokens is in_tokens minus that floor. | Unlike a cell's tokens, which are re-resolvable from the proxy DB afterwards, a baseline has no stored window to re-resolve from — all 48 cells would have lost the headline cost metric permanently. Recovered here only because the floor is a property of the combination rather than of any question, so it could be re-measured after the fact and is disclosed as post-hoc on every row it touched. | | 26 | The report never stated the terms it was measured under. run.json had carried the continuation budget since the feature landed; the report never read the field. | The one term that makes two runs incomparable was absent from the document a reader compares runs with. Now on the second line, beside the commit and the model. | | 27 | A warning asserted a cause it had not established — twice. The token-ambiguity note first said another session of that agent runs alongside the benchmark; re-run those cells on an otherwise idle machine. The machine was idle, so that remedy changed nothing. It was then rewritten to blame the previous cell bleeding across the window boundary, with arithmetic offered as proof: one flagged cell's 274,139 tokens being "its own 139,727 plus its predecessor's 134,412". | That was wrong too, and the arithmetic is what made it convincing. The 134,412 session was the same cell's first attempt — the cell had failed once and been retried. The actual predecessor was a third session of 172,223 tokens that was never counted. Across the whole run no session starts inside more than one cell's window, so the bleed never existed. A warning that confidently misdiagnoses is worse than one admitting ignorance, and a second confident misdiagnosis of the same rows is worse still: the second one was believed because it came with numbers. The warning now describes what was observed and names no cause. See defect 29 for the underlying bug, now closed. | | 28 | A flag was parsed, then dropped on detach. The supervisor re-execs itself under nohup to escape the process group, and that relaunch enumerates its flags explicitly. --continuations was added to the parser but not to the relaunch. | The run would have proceeded silently at budget 0 while its log said otherwise. Caught before launch by stubbing node on PATH and reading the argv each pass actually received, rather than trusting that threading a flag through is trivial. | | 29 | A row described its last attempt while its tokens described the whole cell. runCell resolved tokens over a window spanning every attempt, then built the row by spreading the last attempt's result. So a retried cell recorded that attempt's started_at and wall_s beside an all-attempts token total. Three consequences: the row could not reproduce its own number (re-resolving from its own window returns about half, which the offline backfill would have written back as an improvement); wall_s charged a cell that burned 73.6s as 35.6s; and every retried cell tripped the ambiguity check, because a retry is a fresh spawn and opens a session of its own. | The published analysis then excluded those 21 rows as over-counts — and since a retried cell pays for two attempts, excluding them pushed opencode's measured cost down, from 1.38× to 1.16× claude's. A correction applied in the wrong direction to correct data, on the strength of defect 27's confident wrong cause. Fixed at the source: the row now records the cell's span and per-attempt windows, ambiguity is judged per attempt (one session per attempt is a retry, two inside one attempt is an anomaly), and the backfill refuses any window narrower than the cell it describes. The run's rows were repaired offline from the proxy DB with no cell re-run, checked by requiring per-attempt and whole-span attribution to agree to the token. | | 30 | A wall-clock sum dropped its middle legs. The continuation loop computed wall_s as first + last, which is exact at a budget of 1 and lossy at 2 — the value the repository had just adopted as its default. | Found while fixing 29, not by a failing test, because no test exercised the continuation loop's arithmetic at all. It also blocks repairing the budget-2 run the same way: with attempt 1's duration under-recorded, the walk-back that reconstructs earlier attempt windows lands too late, and the repair script's controls refuse all four of that run's retried cells rather than writing a plausible wrong answer. | | 31 | A published figure was arithmetically impossible, and nobody multiplied it out. The budget comparison quoted a shared-denominator mean of 0.935 for budget 1. That run answered 44 of 48 cells with a score sum of 43.00, so the mean over 48 is 0.896; 0.935 would require those 44 cells to average 1.020, above the maximum score. | It survived because it sat between two figures that were right (0.977 over answered, 44/48 answered) and pointed the way the surrounding prose already argued. A number that agrees with the argument does not get checked. The correct value makes the budget look BETTER than the retracted claim — 0.896 → 0.975 rather than 0.935 → 0.948 — so the error was not motivated, merely unverified. The claims checker now recomputes it from the rows rather than matching the text. | | 32 | A trade-off was published from one run's noise. The budget was reported to buy completion at the cost of quality, mean score over answered cells falling 0.977 → 0.948. Re-running the same 48 cells at the same budget on a corrected harness gives 0.975 — no fall. | The claimed effect was −0.029. Between two runs identical in arm, agent, model, budget and questions, single questions move the 48-cell mean by −0.011, +0.021 and +0.018. The effect was never larger than the noise, and it was published as a candid admission of a cost — the kind of claim that invites no scrutiny because it argues against its author. Per-question figures here need a replicate before they mean anything: within ONE run, a question's content tokens vary across its 3 reps by a median factor of 1.5× (claude), 1.7× (copilot), 1.9× (opencode), worst observed 12.5×. | | 33 | A two-value question was read as an arm difference. A4 scores either 0.82 or 1.00 and nothing else, so a three-rep median is decided by which value lands twice. In r8 all four arms produced both values and the medians split 2–2; the page reported that graphify and hybrid "drop A4". | Pooled over three runs the arms sit at 0.78–0.89 with hybrid highest, and r7 alone at ten reps per arm puts all four at exactly 0.82. The tell was visible in the run's own data and never looked at: a per-question median is only meaningful if the underlying cells are not bimodal, and printing the distinct values per question would have shown A4 taking two. The claim is withdrawn, along with the neater sentence it supported — grep is 1.00 on all sixteen questions in r8, but scores 0.82 on A4 in r7 and x2. | | 34 | A null result was published as evidence because it pointed somewhere flattering. Zero hallucinations in the forced graph arms was called "the one result that favours an index", hedged as too small to lean on, and then leaned on. | Balanced claude-only across four runs it is 2/72 against 0/72, where the expected count under a shared rate is 1.0 and P(observing zero) = 0.37. Per-run counts are 0, 0, 1, 4 — r8 is the outlier that made the pattern visible. Both claude hallucinations are grep's while hybrid, also text-search, has none, so the framing fails inside its own family. Detecting a real 1.4% difference needs ~400 abstain cells per family against the 72 available. Hedging a claim is not a substitute for testing it: the hedge was accurate and the claim was still repeated in three places. | | 35 | A supporting citation matched no run in the corpus. The tool-choice section cited r6 as "4 graph calls in 348" as one of three replications. Scanning every (run, arm, agent) combination, nothing produces 4/348; r6's hybrid arm is 3 in 322, and the nearest 348-ish figure is r6's grep arm at 0/350 — an arm with no graph tools at all. | It survived because it was corroborating evidence for a conclusion that is, as it turns out, correct: pooled over four runs the rate is 1.57% with every run inside Poisson noise. A wrong number in support of a right answer is the hardest kind to find, because the conclusion it serves keeps passing review. The fix is structural rather than careful reading — the claims checker now recomputes the pooled counts from results.jsonl across all four runs, and refuses to pool a run whose hybrid tool surface differs. | | 36 | The replication check itself read medians, and a median hid a cell. The audit that withdrew A4 and the hallucination result graded each per-question claim on its per-run medians. On that basis B3 was published as an r8-only artifact. Counting cells instead, CodeGraph also fails a B3 cell in x2 — the minority value of a 1.00 / 1.00 / 0.00 triple, invisible to the median. The same recount showed Graphify missing 5 of 16 A1 cells across two runs, a result no version of this page had mentioned. | The audit was written to catch exactly this failure and then committed it, one lesson late: "a class median hides a bad question" and "a per-question median hides a bad cell" are the same defect at two scales, and only the first had been internalised. Per-question verdicts are now counted in cells across every run sharing the answer key, and the claims checker computes those counts rather than pinning any median. | | 37 | A keyword-in-answer metric counted denials as citations, and a published verdict was built on it. CodeGraph's A1 losses were reported as a separate, unexplained problem on the strength of /docker-compose/i.test(answer): the arm "cites the file in 10 of 16 cells and still scores 1.00 in only 4 of those 10", therefore "reaching the file is not its problem". Six of those ten cells name the file only to say they could not read it — "the real 'coding' repo isn't checked out here". Reaching the file was the entire problem. | A substring test cannot tell an assertion from its negation, and the direction it got wrong was the direction the conclusion turned on. Re-classified by what the answer claims rather than which words it contains, the axis is perfect: 29 of 29 cells claiming to have read the file score 1.00, and 0 of 5 claiming the repository is absent do. The checker now pins the claim classification, keeps the 4/10 artifact relabelled so it cannot be cited again, and verifies against run.json that the repository was in fact present. | | 38 | The regex written to fix defect 37 committed defect 37, one level in. Replacing the substring test with a phrase-enumerating one — a list of the ways an answer might say it could not read the file — produced "five failing cells make no claim in either direction" and a cross-question total of 7. Reading all sixteen answers gives two silent cells and at least 18 denial cells: the pattern missed r7 rep8's "no .codegraph/ index or file access available in this sandbox", a denial phrased in words it had not listed, and scored two memory-attributed cells as silent. | Enumerating phrasings is the same defect as matching a substring: both decide a semantic question with a lexical test, and both fail silently by under-matching rather than loudly by erroring. The categories are now hand-audited from reading all sixteen answers end to end and pinned cell by cell; the regex survives only as a tripwire on the hit count, so changed data forces a re-read instead of letting a stale hand-audit describe it. The one cross-question figure still regex-derived is labelled a lower bound on the page. | | 39 | The CodeGraph arm's index never covered the tree the benchmark was testing. Arms run in a worktree under os.tmpdir(); the container mounts only ${HOME}/Agentic, so that worktree is invisible to the container-side MCP server, whose default project (/coding) is uninitialized. At least 30 of 172 CodeGraph cells report the index unreachable, and when it does answer it serves /workspace/coding — the main working tree, which is not the de-contaminated one every other arm searches. 15 of 31 sandbox-excluded paths are present in it (not the answer key, not the observation exports). | The sandbox verifies containment of the tree it builds, and the codegraph arm reads an index of a different tree, so the guarantee never applied to it. It surfaced only by chasing five L2 zeros that turned out to be correct refusals. Split by whether the index answered, L2 is 0.00×5 (harness) and 0.50×4 (capability) rather than a single 0.22 — the capability result is cleaner than the blend it was reported as. FIXED (b48c2d38e, fddf1aacc) and re-measured as coding-v1-r8-cgidx: the index is now built over a second worktree of the same commit with the same exclusions, and the MCP server is pinned to it. Unreachable-index cells 14 -> 0; L2 0.17 -> 1.00; A1 unchanged, which confirms corpus scope as a separate cause. The larger finding is behavioural — CodeGraph's MCP share went 14% -> 59% — so the earlier numbers measured an arm degraded to Read with a broken tool attached. | | 40 | A correct abstention was scored as a hallucination because its subject phrase was four characters too long. Hybrid's T3 cell says "No payment-processing/financial-transaction reconciliation module exists in this codebase" and was graded 0.00 with hallucinated: true. The matcher allows sixty characters between no and exists; this subject spans sixty-four. A second abstention in the same answer was missed too, because \bno (?:module|…) requires the noun immediately after no and a quoted subject sat between them. | The window was tuned on the answers that existed when it was written. It is the same defect as Lesson 3's decoration mismatches, one layer up: not a character class but a LENGTH. Worse, the failing cell was the only one of three to call a graph tool — the graph surfaced adjacent real files, the arm named them while ruling them out, and the longer sentence overran the window. A working tool made the answer better and the grader punished the extra words. Counted properly hybrid's T3 is 1.00. FIXED: the gap between the negator and its head is now counted in WORDS, not characters, which also subsumes the adjacency defect. Regraded from stored answers across every run in the corpus; it moved exactly three cells — r8 grep/T4, x2 grep/T3, r8-cgidx hybrid/T3 — each 0.00 -> 1.00 with a false hallucinated cleared. The page's hallucination count drops 4 -> 3 and grep's pooled drop-list loses T3. | | 41 | Four runs were carrying scores from a superseded answer key, and a blind fix would have corrupted a fifth. Reconciling every run against the current grader found 105 stale rows. Fourteen in r5 and fourteen in r6 were L2 and A1 scored with keys since corrected — L2's key had named the wrong file, so every arm scored 0.15 on a correct answer. Six in replication-full were S1 cells that predate the contamination detector and cite the benchmark's own ground truth; they are now void rather than scored. | The remaining 64 rows were the trap. They were r5/r6 cells for A3, A4 and B1 — questions whose PROMPTS were rewritten by c31d07b02 after those runs executed. Their stored answers respond to different words, so re-scoring them with today's checklist is not a correction; it is a category error that looks exactly like a correction, because it produces plausible numbers on real data. A one-line regrade would have quietly rewritten a third of r5. The fix is structural: kgbench-regrade.mjs now reconstructs the question set from the run's own commit and refuses any question whose prompt moved, and refuses an entire run whose set cannot be recovered — which is what retires the two pilots, whose question file was never committed. Cannot verify is not verified. PARTLY RESOLVED: the 64 refused rows have been RE-MEASURED rather than regraded, as coding-v1-r5-requestions and coding-v1-r6-requestions at each run's own corpus commit — beside r5/r6, not inside them, because those runs recorded no model and splicing would confound arm with model on exactly the questions under test. r5/r6's own A3/A4/B1 columns remain those of a superseded question set and always will; that is a fact about the runs, not a defect left open. | | 42 | A 121-cell run was killed twice, at cells 2 and 18, by the tool that launched it. Neither death produced an error: the log simply stopped, supervise.status stayed running, and no process remained. The cause was the launching harness's 10-minute call ceiling killing the process group it hosted — kgbench-supervise.sh detaches, but detaching does not leave the group. | A supervisor that survives signals does not survive its own parent's death, and the two failures are indistinguishable from the outside. The tell is status: running with zero live processes, which is why the monitor now asserts BOTH rather than either. Re-hosting under launchctl submit fixed it. Nothing was lost at any point — the runner writes cells incrementally and resumes from results.jsonl, so the restart continued at cell 18 rather than redoing 18. Incremental writes are what turned two silent kills into a delay instead of a loss, and that property is worth more than the supervisor's retry logic. | | 43 | Four runs of cost figures described a defect rather than a backend. r8 reported CodeGraph at 161,322 tokens and 50.3s — 2.1x grep's tokens and 2.8x its latency — and the page built its headline economic claim on it: "both cost 1.7-2.1x the tokens, 1.6-2.8x the latency". With an index that actually covers the corpus, the same arm on the same questions runs at 1.05x grep's tokens and 0.93x its latency. Its median tool calls on L2 fall from 12 to 2. | A broken retrieval tool does not look broken in a cost table; it looks expensive. The arm compensated by reading files, and reading files is what the tokens measured. Nothing in the correctness column flagged it, because the arm still answered correctly — it just paid Read prices to do it. The tell was in the tool mix, which no published table showed. A benchmark that reports what an arm COSTS without reporting what it DOES cannot distinguish an expensive backend from a broken one. | | 44 | The agent-axis table was r8's, on a page whose header declares r9. All six rows matched r8 exactly — including the giveaway hybrid/opencode count of 46/48, which exists only in r8 (r9 is 44/48). The prose built on it — "copilot costs 1.85x claude's content tokens on the identical arm, and opencode 1.38x" — is r8's grep-arm ratio. In r9 those are 1.11x and 1.71x: the two agents swap places, and the section's old heading (the agent axis is larger than the arm axis) inverts with them, because r9's arm spread is 2.02x against an agent spread of 1.71x. | The checker held the right numbers and never compared them to the page. kgbench-verify-report-claims.mjs already hardcoded r9's values and asserted opencode ~1.71x and copilot ~1.11x — and passed, because it checks its own constants against results.jsonl and never reads a number OUT of the rendered table. Its docstring claims those constants are "a snapshot of what the published page asserts"; for this table they were a snapshot of what it should have asserted, and the gap between those two is exactly the defect. A generated table carries no run label, so a stale paste is indistinguishable from a fresh one — both are plausible, and this one survived a full rewrite of the surrounding prose. FIXED: the table and its ratios are r9's, and the checker now PARSES the six rows and both ratios out of README.md and compares them to the run, so a table from the wrong run fails on sight. A second failure mode is closed with it — the checker read docs/benchmarks/coding-v1/report.json, which the NEXT published run overwrites, so publishing the 128-cell run on 2026-08-19 turned four checks red against a page that had not changed; it now reads the run's own preserved report.json. |

Defects 1–5 all pointed the same direction — flattering the graph arms, penalising grep. Defects 7 and 9 point the other way. Defect 10 flattered nobody and hid everybody. Defect 15 manufactured a capability finding out of a termination bug, 16 destroyed the explanation of all the others, and 17 quietly charged every cell part of its neighbour's bill. The lesson is not "the graph arms were flattered", it is that every measurement defect found here was invisible in the output it produced. Each one yielded a clean-looking table.

Defect 18 deserves its own line, because it is the failure mode of fixing things. It made a correct fix look inert — the ambiguity count read 94 before and 94 after — by leaving the old verdict in place beside the new evidence. A stale field that contradicts a fresh one is worse than either a wrong answer or no answer, because it argues against the repair that just succeeded. The same shape as defect 15, in the tooling rather than the data.

Defects 19–28 sharpen the same lesson rather than adding a new one, and three of them are worth separating out.

19 and 20 are the fifth and sixth leak in a series where every one was a comment explaining the previous leak. The pattern held right up to the fix: documenting that a sentence frame leaks required quoting the frame. Editorial discipline has now failed six consecutive times at the same task, so the control is no longer editorial — a scan runs against the harness's own source and fails the run.

22 and 23 are the same shape as 15: a harness artifact wearing a capability finding's clothes. x2 published opencode at 13% completion and a median of 1.00; both numbers were artifacts. The completion rate measured a turn budget nobody had equalised, and the 1.00 was survivorship over the 13% of cells that happened to write. Correcting the first lowered the second, because answering more questions means answering harder ones. An agent that looks uniformly terrible, or uniformly perfect on a small denominator, is a hypothesis about the harness before it is a finding about the agent.

24 and 27 are both failures of a confident wrong answer over an honest absent one. A pin that logs itself as applied and is then discarded downstream is worse than no pin, and a warning that names the wrong cause is worse than one that says it cannot tell — in each case the reader stops looking, which is precisely what the mechanism was supposed to prevent.

Most were found by instrumentation rather than by reading results: the tool-surface check, the containment scan, the orphaned-MCP-server guard, and the grader's own disagreement counter, which exposed defects 11, 12 and 13. Defect 15 was caught by neither — it was caught by asking why a plausible-looking 0.00 was so uniform, and then counting distinct answer texts. An agent that fails everything is not a finding; it is a hypothesis, and the cheapest test of it is whether its answers are even different from each other.

A note on tuning the grader after seeing results¶

Scoring fixes across these runs changed rows that had already been graded, which is exactly the shape of a result being massaged. What makes it defensible, and how to check:

• Every change was validated against fabrication fixtures that must still be caught, not just against the rows it fixed.
• Every change was applied by re-grading all cells uniformly, never one arm.
• The full diff is committed: results.pre-regrade.jsonl holds the original scores and regrade.json lists every row that moved.

When the machine lied¶

Worth its own note, because it is easy to miss. Node timers cannot fire early, so a 300s timeout completing at 950s is proof the process was starved, not that the work was slow. Recorded naively, that becomes hard_fail_rate — a permanent, published claim that an arm cannot answer a class of question, caused entirely by an antivirus scan.

Reproduce it¶

# check every arm is available (fails loudly if an index or the proxy is missing)
node scripts/kgbench-run.mjs --set coding-v1 --preflight-only

# the full matrix, detached and self-resuming — USE THIS for anything long
scripts/kgbench-supervise.sh --run-id my-run --set coding-v1 --reps 3 \
                             --agents claude,copilot,opencode

# progress / outcome
cat .data/kgbench/runs/my-run/supervise.status
wc -l .data/kgbench/runs/my-run/results.jsonl

# re-apply a fixed grader to stored answers, without re-running the matrix
node scripts/kgbench-regrade.mjs --run my-run --dry-run

# render the generated tables and the figures
node scripts/kgbench-report.mjs --run my-run --out docs/benchmarks/coding-v1/RESULTS.md
node scripts/kgbench-charts.mjs --run my-run --agent claude --out docs/images

# does the prose on this page still match the data it describes?
node scripts/kgbench-verify-report-claims.mjs

RESULTS.md is generated; this page is not. kgbench-report.mjs --out refuses to write to a file lacking its generated marker, so it can no longer replace this analysis with the machine version — that happened twice before the guard existed (defect 16). After a re-render, update the numbers quoted on this page by hand.

Pass --agent to the charts on any multi-agent run. Without it every bar pools agents whose tool enforcement differs, which is the comparison this report marks as not meaningful; the script warns when you do.

Full answers are stored, so a fixed grader can be re-applied offline instead of re-running the matrix.

Files¶