Benchmarks
The numbers, and exactly how we got them.
One measured run, published with its method and its caveats. Where a benchmark hasn't been run yet, this page says so instead of showing you a chart.
A run happened, on named hardware, against a named corpus, with a reproduction command below. The 10.7× context table, the search-quality ladder, and the ANE fix on this page are all this category. Numbers here can go stale as the code changes; they cannot be wrong about the run that produced them.
A mechanism is specified, and often a real incident motivated it, but nothing has been benchmarked yet — so nothing is quoted as a multiplier. Workspace leases, burst coalescing, checkpoint/restore, and Anvil Pods on the home page carry this label. Where we cite a reference number from adjacent work (see the pod sync note there), we say whose workload it was measured on.
Corpus: the Ganvil repository itself at commit 455bb84c5c57 (135 files, 88 Rust), imported via gan import and served by gan mcp from the imported store. Run 2026-07-08.
A driver speaks MCP over stdio and issues a realistic mix of ten tool calls, recording each response's size. Each call is paired with a counterfactual: the bytes the equivalent read-grep-read workflow ingests, measured from the actual repository. For callers, that means the summed size of every file containing the queried name — which is what "grep, then read the hits" costs. Tokens estimated at 4 characters per token on both sides.
The counterfactuals are deliberately conservative. They exclude wrong-file reads, retries, and re-reads across turns, and they assume the grep workflow finds the right files on the first pass. Real agents don't.
| Tool call | Response tokens | Counterfactual tokens | Saving |
|---|---|---|---|
| outline (repository.rs) | 1,738 | 4,814 — read the file | 2.8× |
| get_symbol (one function body) | 889 | 4,705 — read its file | 5.3× |
| callers (resolve_rev, depth 1) | 141 | 14,820 — grep + read 8 hit files | 105× |
| impact (transitive + tests) | 2,012 | 15,210 — grep + read hit files | 7.6× |
| search (ranked, degraded mode) | 406 | ~5,700 — grep + read candidates | 14× |
| structural_grep (.unwrap() in ganvil-core) | 2,012 | ~22,200 — grep + read to filter false positives | 11× |
| context_pack (task bundle, 2k budget) | 2,012 | 26,420 — read top 6 relevant files | 13× |
| symbol_diff (HEAD vs worktree) | 14 | ~2,000 — read the text diff | 143× |
| blame_symbol | 74 | ~5,000 — read log -p of the file | 68× |
| log_search | 283 | ~1,500 — read full log output | 5.3× |
| Total | 9,581 | 102,410 | 10.7× |
~90% of context-acquisition tokens disappear.
A session that spends 100k tokens rediscovering the repository needs about 10k through the tool surface.
Graph queries are the outliers.
callers at 105× and symbol_diff at 143×: the response is a list of names where the counterfactual is many whole files. These are also the queries agents repeat most.
outline’s 2.8× is the floor.
Measured on a file that is mostly API surface; body-heavy files fare far better.
The multiplier grows with repo size.
Responses are roughly constant-size; counterfactual file reads grow with the codebase. The benchmark corpus is small (135 files) — large repos sit well above 10.7×.
Real-world savings sit above these figures.
Counterfactuals exclude retries and wrong-file reads; deterministic tool output turns repeat queries into prompt-cache hits; and context_pack replaces the whole multi-turn exploration phase in one call.
Retrieval behind search is a three-rung ladder: lexical ranking (the zero-AI rail), hashed n-gram vectors, and a real on-device embedding model (all-MiniLM-L6-v2, ONNX). An 18-query gold set — paraphrases, not verbatim identifiers, because verbatim lookups would flatter the weaker tiers — runs against the real serving code path of each tier. A tier scores a hit when a gold file appears in its top 10. ms/query is mean wall-clock per query from the same run. Run 2026-07-10 on an Apple A18 Pro (8 GB RAM); corpus is the Ganvil repository itself (230 files).
| Tier | hit@1 | hit@5 | hit@10 | MRR@10 | ms/query |
|---|---|---|---|---|---|
| lexical | 0.22 | 0.56 | 0.61 | 0.360 | 134.5 |
| hashed | 0.11 | 0.50 | 0.61 | 0.277 | 0.3 |
| model | 0.61 | 0.89 | 1.00 | 0.735 | 26.6 |
The ladder’s payoff is the top rung.
The on-device model is the only tier that reliably finds things from a paraphrase: hit@1 0.61 against 0.22 (lexical) and 0.11 (hashed), and every query lands a gold file in the top 10 — at 26.6 ms per query.
The middle rung buys latency, not quality.
Hashed n-gram vectors score below plain lexical ranking on paraphrase queries (hit@1 0.11 vs 0.22, MRR 0.277 vs 0.360). What hashing buys is serving cost: 0.3 ms per query against lexical’s 134.5 ms. It keeps search cheap before the real embeddings exist — it does not make it smart. We measured it; we’re telling you.
Lexical pays its cost on every call.
The zero-AI rail re-scores the whole corpus per query, tree-sitter symbol extraction included — that is its 134.5 ms. Right as a fallback that works with AI off; wrong as a steady state.
One-off cost, then it’s cheap.
Embedding the 230-file corpus took 128.7 s cold on CPU (this run pinned the CPU provider; quality columns are device-independent). The vector cache and background worker absorb it — steady-state queries pay only query embedding plus cosine.
The neural engine is the default embedding device on Apple silicon — first rung on the hardware ladder, ahead of GPU and the hosted hammer tier. Until today, bulk corpus embedding on that default path did not finish: we watched it sit hung for over ten minutes before giving up and pinning the CPU provider, which is the workaround baked into the search-quality run above.
The cause was shape churn. CoreML compiles a fresh graph for every distinct tensor shape it sees, and batching by sequence length alone still let the batch-size axis roam free — up to 128 shape combinations for one corpus, each triggering its own compile. The fix buckets both axes into a fixed set of 12 shapes total, compiled once per process. Same machine, same default device, same corpus: full coverage in 5 seconds.
We run this on our own machines, building this exact platform — the same continuity as the dependency-mirroring field report on the home page. That is also why we noticed: the wedge first showed up mid-benchmark, not in a lab.
# token savings cargo build -p gan gan import <any-git-repo> --as /tmp/corpus && cd /tmp/corpus # drive `gan mcp` over stdio with initialize + tools/call frames; # compare response sizes against the summed sizes of the files the # equivalent grep-and-read workflow touches. # search quality (writes docs/benchmarks/search-quality.md) cargo run --release -p ganvil-mcp --example search_eval --features eval-real-model
The methodology is in the repository at docs/benchmarks/2026-07-08-mcp-token-savings.md and docs/benchmarks/search-quality.md. If your run disagrees with ours, we want to hear about it.