tool-selection.md — data appendix¶

Generated by scripts/analysis/tool-selection-analysis.mjs. Do not hand-edit.

Runs analysed: coding-v1-r6, coding-v1-r7, coding-v1-x2, coding-v1-r8, coding-v1-r9. Arm: hybrid unless stated.

1. Tool telemetry by agent¶

A run with 0 recorded calls is not a run with no tool use — it is a run with no instrumentation. Every later section is therefore claude-only.

2. Graph uptake per run¶

unanimous counts questions whose reps all agree (all used a graph, or none did). It stays at 14–16 of 16 in every run: tool choice is a near-deterministic function of the question rather than a rate.

3. Did uptake really rise from r8 to r9?¶

The cell-level test treats 48 correlated cells as independent and overstates its own precision. Clustered on the question the shift is directionally clean — nothing lost the graph — but not resolvable at n=16.

Per-question detail¶

4. When is the choice made?¶

First executed tool per cell. In both runs exactly one cell of 48 begins with text search and later reaches for a graph — the agent commits on turn 0 and does not revise.

coding-v1-r8 — opened with a graph tool: 5; switched to one later: 1

coding-v1-r9 — opened with a graph tool: 17; switched to one later: 1

5. Executed-tool histogram — coding-v1-r9¶

6. Cost — the cross-cell comparison and the paired one¶

Comparing cells directly appears to vindicate the preference decisively:

That comparison is confounded: the graph is called on a stable subset of questions, so this compares questions rather than strategies. The unconfounded version restricts to same-run, same-question pairs where some reps used a graph and some did not.

Paired comparisons available: 11.

The interval spans 1.0 and the sign test is nowhere near significance. Controlling for the question, the token difference between reaching for a graph and not reaching for one is indistinguishable from zero — the graph is neither reliably cheaper nor reliably dearer. That is the honest reading, and it supersedes an earlier n=8 pass of this same comparison which gave 0.81× and was written up as a reversal. Three more pairs moved the mean to 0.94× and widened the interval across 1.0; a mean of eight ratios quoted without dispersion read as a result it never was.

What survives is the negative half: the cross-cell table above is confounded by question difficulty and cannot support the claim that using a graph costs 2.29× more. That inference is retired. It is NOT replaced by the opposite one.

7. Are the questions keyword-addressable?¶

For each question, a literal token appearing verbatim in its own prompt is searched across the repository and the hit set checked against that question's ground-truth evidence paths. git grep -l is used, so "rank" is path order, not relevance — the claim being tested is reachability, not ranking.

13 of 14 gradeable non-abstain questions have their ground truth in the hit set of a single literal grep of a token the question hands the agent. This is a benchmark of keyword-addressable retrieval, and grep is the optimal instrument for that.

8. Is the answer even in the index?¶

Graphify index at commit b6ed72ff68894caa077fe08f49112105afe8182a: 58124 nodes over 4347 distinct source files.

17/25 ground-truth evidence files are present in the Graphify index. 7 of 14 questions have at least one required fact the graph cannot see at any price.

Does coverage drive the choice? (no)¶

No difference. The agent is not avoiding the graph where the graph is blind — per section 4 it commits before it could possibly find out.