NeuralMind — the honest public benchmark

What this is: a reproducible, no-cherry-picking comparison of how much context different approaches put in an agent’s window to answer a real code question, and whether the objectively-correct file actually makes it in. Cost (tokens) and correctness (gold-file recall) are reported together — a token-reduction number with no correctness number attached is meaningless.

Reproduce it yourself (no trust required):

git clone https://github.com/dfrostar/neuralmind && cd neuralmind
pip install -e . tiktoken           # source checkout — ships the evals/public harness
NEURALMIND_ORT_THREADS=1 python -m evals.public.run   # clones the pinned repos, prints the table
# or, from the clone: neuralmind benchmark --public

The benchmark harness (evals/public) ships in the source tree, not the PyPI wheel, so run it from a clone — pip install neuralmind alone won’t have it. The pinned repos are cloned at fixed commit SHAs and nothing in the run is random, so gold-file recall and found-rate should match the tables below exactly; they have on every machine we’ve compared. Token counts can differ slightly between machines: for the previous snapshot, runners without AVX-512 landed up to 0.8% off on a per-repo mean. The v4.12.0 tables below come from an x86-64 machine with AVX-512 and have not yet been compared against one without it. NEURALMIND_ORT_THREADS=1 matches CI’s configuration (it applies to neuralmind benchmark --public too). What moves, and the evidence: How exactly a re-run reproduces.

Method (designed to be hard to dismiss)

The baselines

• full-file — paste every source file into context. Recall is trivially 1.0; cost is the whole repo. This is the honest ceiling the “just give the model the files” workflow actually pays.
• ripgrep — extract query keywords, rank files by match count, open the top-5 whole files (what “just grep and read the files” costs).
• embedding-rag — the top 8 entries of NeuralMind’s own vector index (same encoder), sending only their indexed text: a symbol’s qualified name, module and docstring, not its code. This is NeuralMind’s vector-retrieval core in isolation, so the neuralmind vs embedding-rag gap isolates what the ranking and assembly add (and cost) on top of raw top-k. Its token count is a floor: a chunk RAG that sends code bodies pays more.
• neuralmind — NeuralMind.query: progressive L0–L3 disclosure assembling a compact, structured context (project map, the query’s other candidates, and eight ranked hits), counted with the same tokenizer as every other backend.

Scoring reuses neuralmind/quality.py verbatim — the same metric code the CI quality gate runs — so these are not numbers invented for a press release.

Results

Tokenizer: tiktoken o200k_base. Generated by python -m evals.public.run; raw per-query data in bench/public/results.json.

requests @ 0e322af877 — 14 pre-registered queries

No NeuralMind gold-file misses on this repo.

click @ 874ca2bc1c — 7 pre-registered queries

No NeuralMind gold-file misses on this repo.

flask @ c12a5d874c — 10 pre-registered queries

No NeuralMind gold-file misses on this repo.

rich @ 7f580bdcf0 — 9 pre-registered queries

No NeuralMind gold-file misses on this repo.

Aggregate across all 4 repos (40 queries)

Every one of the 40 queries retrieved its gold files (40 of 40, found-rate 100% on each repo), at 51.2×–242.0× fewer tokens than pasting whole files. The token ratio is not uniform across repos — that’s the honest picture, not a single cherry-picked ratio.

Forty queries is a small sample, and a clean sweep on it is not a claim that NeuralMind never misses. On the 150-question retrieval eval (bench/retrieval), the same build puts a gold file in the top five for 93.3% of questions; the misses are listed there, question by question. Every earlier run of this benchmark had at least one miss, and the next code change can bring one back.

What the numbers honestly say

1. Against what developers actually do today — paste files or grep — NeuralMind is a large, real win, but not a perfect one. It finds the gold file on all 40 queries at 51.2–242.0× fewer tokens than pasting the files. Pasting every file is the easiest baseline to beat, so weigh the next one more: against ripgrep opening its top five files, NeuralMind uses 31.2–67.2× fewer tokens (requests 32.5×, click 67.2×, flask 31.2×, rich 45.2×), and ripgrep finds the gold file on 32 of 40 queries to NeuralMind’s 40. On recall it’s ahead on 3 of 4 repos (requests, click, flask) and ties on the other one (rich), never behind. The ripgrep baseline still opens whole files: an agent that greps and then reads only the matching lines pays less than it, and that workflow is not a baseline here. Most agents don’t have a tuned vector index sitting there; they read files or grep. That is the baseline NeuralMind replaces. It is not a claim of zero misses — see “Where NeuralMind loses” below.

2. Plain vector search is excellent at findability, and we show it. embedding-rag is NeuralMind’s own vector index with nothing added, so this comparison isolates what NeuralMind’s ranking and context assembly add.
   • Recall: NeuralMind is now level with it on requests, click and rich, and ahead on flask (1.00 vs 0.85). Across all 40 queries it is 40 of 40 to the baseline’s 38.5.
   • Before v4.12.0 it was the other way round: the baseline matched or beat NeuralMind’s recall on every repo, at 98.75% to 95%. The ranking layers were discarding gold files that vector search had found. See the v4.12.0 release notes.
   • Rank: the baseline still places the gold file higher on flask (MRR 0.74 vs 0.72) and rich (0.94 vs 0.83).
   • Cost: the baseline’s token count is a floor, not what a vector RAG pays. It sends only each entry’s indexed text: a symbol’s name, module and docstring, never its code. NeuralMind’s tokens buy the assembled context an agent uses to answer: the project map, eight ranked hits, and the query’s other candidates.
3. ripgrep is the cautionary tale on 3 of 4 repos. Cheap-ish, but on requests and click it misses the right file 21–29% of the time (recall 0.79) — keyword search has no notion of meaning. It only reaches 1.00 recall on rich, where the query vocabulary happens to overlap the code closely.

Where NeuralMind loses

Three loss modes, reported plainly:

• Misses outside these 40 queries. This run has none, but the 150-question retrieval eval misses the gold file in the top five on 10 questions, most of them on rich and on this repository. On this repository four of its six misses are vector-search misses: the gold file’s best entry ranks 23rd to 39th, past the 20 candidates NeuralMind re-ranks. The other two are in the pool, but five other files rank above them. The three v4.10.0 misses here (xfile-status-codes, echo-util, xfile-dispatch-context) and request-wrapper, missed by an earlier v4.12.0 build, are found now. These numbers move between code changes; see “The corpus” below. Full per-query detail is in the tables above and in bench/public/results.json.
• Rank quality against bare vector retrieval. On flask and rich, the baseline ranks the gold file higher. The re-ranking NeuralMind adds still costs some rank on those two repos.
• Token cost against bare vector retrieval. NeuralMind spends 3.0–4.1× the baseline’s tokens to deliver assembled context. If your only need is “which file”, a bare vector index is cheaper.

What this benchmark does not measure (on purpose)

• The synapse learning layer. Synapse weights are session- and usage-dependent — they learn from how you work — so they cannot be part of a fixed, reproducible public number. Injection is OFF here. The learning lift is measured separately and reproducibly by the synapse A/B eval (tests/benchmark/run.py, Phase 2): +3.5 to +14 points top-k hit-rate across runs, CI-gated on direction, budget-neutral, on the ~500-line reference fixture. That shows the direction, not that the layer pays off on a real codebase over weeks of use: nothing published shows that yet. neuralmind eval . --synapse-ab (v4.14.0+) measures it on your own repo, with your own questions, and records whether the gap grows as the layer does (walkthrough).
• A repository indexed from its root. Each repo is indexed from its source directory only (the manifest’s subdir, e.g. src/click), so its tests, examples and docs never compete with the library code for a slot. Indexed from the root, as neuralmind build . indexes a checkout, they do. The multi-repo retrieval eval measures that case on the same four repos and commits (python -m evals.retrieval.run --full-repo, v4.12.1+; raw output in bench/retrieval/examples-v4.12/).
• Task outcomes. Nothing here, or anywhere in this repository, measures whether an agent fixes a bug faster or makes fewer mistakes with NeuralMind. Every number on this page is retrieval: what reaches the window, and whether the right file is in it. A small window can also leave out a caveat a whole file would have shown; only an outcome benchmark (real tasks, solved with and without, timed and checked) would show whether that costs anything. None has been run.
• End-to-end answer quality. Gold-file recall is deliberately a findability metric (objective, no judge). The opt-in answerability arm (--judge) adds the answering signal — see below — but it stays a clearly-labeled secondary, never the headline.

How exactly a re-run reproduces

The machine-to-machine evidence below is for the v4.10.0 snapshot. The v4.12.0 snapshot was generated on one x86-64 machine with AVX-512 (a 4-core container); the drift job will show whether other runners land on it.

Nothing in the run is random: synapse injection is off and no step samples, and every same-machine re-run we’ve done has been byte-identical. Across machines, recall and found-rate have not moved in any run we’ve compared, but token counts sometimes have. Only the two embedding-backed backends, embedding-rag and neuralmind, are affected, which points at floating-point results in the embedding path differing slightly between machines: a tiny difference can reorder near-tied candidates, which changes which chunks fill the context and, now and then, the rank at which a gold file appears.

Which machines land where. On GitHub’s hosted runners, whether the CPU has AVX-512 decides it. The public-benchmark-drift job logs its runner’s CPU, and in all 30 of its runs on main from the v4.6.0 snapshot’s commit (2026-10-04) to v4.10.0’s (2026-10-07), every runner with AVX-512 (AMD EPYC 9V74 and 9V45, Intel Xeon Platinum 8573C, Intel Xeon 6973P-C) produced one output and every runner without it (AMD EPYC 7763, and EPYC 9V74 machines that didn’t expose AVX-512) produced the other. Neither the runner image (two of them), CPython (3.12.14 and 3.12.15) nor turbovec (1.0.0 to 1.1.2) changed which output a run produced, before or after the code change that moved the tables (v4.8.1, see The corpus). The committed results.json is the AVX-512 output, run 37577515538 at d1c538cd (the same retrieval code as v4.10.0). An Apple M3, an ARM chip with no AVX-512, reproduces it byte for byte (below).

What a runner without AVX-512 measures, against the committed run (run 37581970831, v4.10.0’s release commit 17aa6cb2, AMD EPYC 7763):

Mean tokens/query for the (repo, backend) pairs that differ:

CI gates drift, not exact reproduction. The public-benchmark-drift job re-runs this benchmark on every PR and push to main, fails if any repo’s recall moves more than 5 points or its mean tokens/query more than 10% from the committed snapshot, and uploads its fresh results.json as an artifact. (v4.12.0+) It also fails if NeuralMind’s recall on any repo falls below the embedding-rag baseline’s, or if the pooled MRR gap to that baseline widens by more than 0.05 from the committed snapshot. That is the comparison that showed the ranking layers costing recall before v4.12.0; run on the v4.10.0 snapshot, it flags click and requests. The spread above is well inside those tolerances, so the check passes in either state. Both CI runs above used numpy 2.5.3, onnxruntime 1.30.0, tokenizers 0.23.2, turbovec 1.1.2, tiktoken 0.14.0 and NEURALMIND_ORT_THREADS=1. Artifacts are kept for 90 days, which is why the numbers are copied here. The self-benchmark has a recorded history of the same AVX-512 host dependence (token totals differing between runners, hit rates identical since a fix in August), in tests/test_benchmark_regression.py.

Apple Silicon (one maintainer machine, not CI). On 2026-10-07 an Apple M3 (macOS, arm64) ran the benchmark at 175849d6, which is v4.10.0’s code, with CPython 3.12.14, the same third-party packages as CI and NEURALMIND_ORT_THREADS=1, in about 8½ minutes. Its results.json was byte-identical to the committed one. On 2026-10-03 the same machine had matched the v4.3.4 snapshot byte for byte, with the thread pin and without it (about 3 minutes unset).

Earlier snapshots. The split predates the CPU log. For the v4.3.4 snapshot, three main runs on 2026-10-03 landed on two states (37117393222 matched the committed file; 37092548906 and 37133642769 matched each other), with per-repo means at most 1.3% apart and recall, found-rate and MRR identical. For the v4.6.0 snapshot, runners without AVX-512 measured up to 1.9% off on a per-repo mean (requests/neuralmind: 878.0 mean tokens against the committed 895.1), more than the 1.3% this page quoted at the time; the 10% drift tolerance let it pass, and nothing flagged it.

Comparing your own run. Add --out bench/public to the command at the top; git diff bench/public/results.json is then empty when your run matches the committed one byte for byte. NEURALMIND_ORT_THREADS=1 matches CI’s configuration. It made no difference on the M3, so treat it as a precaution rather than a requirement, and drop it if you want the faster run. Expect recall and found-rate to match exactly. Token counts, and the embedding-rag baseline’s MRR on click, may match byte for byte or land on the other state above. The spread above is what we’ve seen for this snapshot so far, not a guaranteed bound; a regenerated snapshot needs its own check.

Answerability arm — --judge (opt-in, secondary signal)

Gold-file recall measures locating the right file, not answering the question. The opt-in answerability arm closes that gap honestly:

ANTHROPIC_API_KEY=… python -m evals.public.run --judge   # off by default

For each query it takes the real context each backend would put in the window — whole files for full-file/ripgrep, retrieved chunks for embedding-rag, the compact L0–L3 assembly for neuralmind — and asks a pinned model (claude-opus-4-8) to answer using only that context (it must say “insufficient context” when the window can’t support an answer). A separate judge call grades each answer against the same def-site gold anchor (the oracle_symbol) on a 0–2 scale, plus a grounded flag.

Why it’s hard to dismiss:

• Same prompts, same pinned model, every backend — each is judged on its own real window, so a low-recall window scores low instead of being papered over from the model’s prior knowledge.
• Published provenance — the answerer prompt, the judge rubric, the pinned model id, and every raw transcript (question, context tokens, answer, verdict, rationale) are committed under bench/public/judge/. Re-score it, or swap the model, yourself.
• Off the deterministic path — needs ANTHROPIC_API_KEY, never runs in CI, and the recall table above is byte-identical with or without --judge.

Honest caveats: it’s LLM-judged (not deterministic like recall); a single judge model is one viewpoint (disclosed, transcripts committed so anyone can re-score with another); and it’s a secondary signal — the recall-at-N×-tokens headline stays primary.

The corpus

The corpus spans four pinned repos / 40 pre-registered queries: requests (14) and click (7) — the original headline pair — plus flask @ c12a5d874c (10) and rich @ 7f580bdcf0 (9), added to harden the “you picked easy repos” critique with two more household-name projects. Every flask/rich query’s gold file is the objective definition site of its named symbol, verified with rg against the pinned commit (e.g. class Flask in app.py, class Console in console.py) — pre-registered in evals/public/manifest.json before any run, exactly as the methodology requires (gold first, numbers second). All four repos’ cost+correctness rows above are committed, current numbers from python -m evals.public.run, not a projection.

Note for anyone diffing against an older copy of this page: earlier snapshots showed 1.00 recall on requests and click, 38–85× fewer tokens. That snapshot was generated once, at the commit that first introduced this benchmark, and was never regenerated afterward even as the retrieval code changed underneath it — the query set itself (including every two-file cross-file query, e.g. xfile-status-codes) has been unchanged since that first commit. The numbers above are a fresh run against current code, not a harder corpus.

This page’s numbers moved twice in the same audit that first caught the staleness above: an initial regeneration (branch code) found 85–100% recall with requests/flask as the weakest repos; merging in several months of independent main development before publishing shifted retrieval behavior again, landing on the 85–100% figures committed at the time, with flask as the weakest repo at 0.85 recall. Neither run was wrong — both were real, deterministic measurements of the code that existed at that moment. That volatility, not any single number, is the actual finding: without either CI regeneration on retrieval-path changes or run-to-run averaging, a benchmark snapshot’s shelf life is one retrieval-code commit. Tracked in ROADMAP.md.

Regenerated 2026-09-27 at v4.3.4. The previous version of this page quoted 93.6% mean recall and 5 misses from a re-run whose raw output was never committed — bench/public/results.json still held an older run, so the tables could not be checked against the data they cited. (Its “93.6%” was also the unweighted average of the per-repo means, labelled as a weighted mean.) The tables above and results.json now come from the same run: two independent runs from fresh clones on the same host (x86-64, AVX-512) produced byte-identical output. That shows the run repeats on one machine, not that every machine gets these bytes: some CI runners have come out up to 67 tokens off on individual queries (at most 1.3% on a per-repo mean), with identical recall — see How exactly a re-run reproduces. The movement since the previous tables: requests recovers xfile-send-adapter (recall 0.89 → 0.93); every other recall figure is unchanged, with small shifts in tokens and MRR.

Regenerated 2026-10-01 for v4.6.0. v4.6.0 changed the default keyword index: the turbovec backend’s BM25 list used to hold only document nodes, so the hybrid fusion gave docs a keyword signal code never got; it is now one BM25 index over docs and code. A retrieval-path change makes the previous tables stale by definition, so they were regenerated against the v4.6.0 code (NEURALMIND_ORT_THREADS=1 python -m evals.public.run --out bench/public) and results.json was committed with them. The movement: requests recovers xfile-redirect-auth (recall 0.93 → 0.96) and flask recovers request-wrapper (0.85 → 0.95), but click now misses echo-util (1.00 → 0.86); rich is unchanged at 1.00. Net: 93.75% → 95% mean, 90% → 92.5% found-rate, 4 misses → 3. Why the index changed, and the five other ranking changes that were measured and not kept, is in the retrieval eval: bench/retrieval/README.md.

Regenerated 2026-10-07 at v4.10.0. The tables went stale without a retrieval change aimed at them. v4.8.1 (#587) corrected the code graph the build writes: a call is attributed to the function it’s in, Python relative imports resolve against the importer’s package, and symbols whose ids collided get their own nodes. The graph’s clusters regrouped on all four repos (click’s four became three), and with them the project summary (L1) and the cluster context (L2) the selector builds from them. Every NeuralMind per-repo mean went up, by 17 to 61 tokens a question, and every ratio went down: requests 46.6× → 45.3×, click 121.7× → 115.2×, flask 78.0× → 76.3×, rich 262.1× → 245.2×, and the range 46–263× → 45–246×. The search hits (L3) changed on 5 of the 40 questions; recall, found-rate and the three misses didn’t move. On click most questions carry the same extra 31 tokens: L1 lists three clusters instead of four (18 fewer), and the cluster those questions load now includes the package’s module docstring (about 49 more). The drift gate passed on every run in between, because it allows 10% on tokens, so the published tables stayed stale from 2026-10-05 until this regeneration. The embedding-rag baseline moved with the same graph: its MRR is 0.69 on click (was 0.60) and 0.72 on flask (was 0.70), and NeuralMind’s flask MRR is 0.73 (was 0.72).

Regenerated 2026-10-08 for v4.12.0. v4.12.0 changed retrieval itself:

• L3 keeps 8 hits chosen from 20 candidates, where it kept 4.
• An intent detected from the question no longer re-weights hits, and the code-signal boost is off.
• Keyword search ignores question words.
• Nodes embed as their name, module and docstring.
• L2 lists the query’s other candidates.
• L3 spreads its eight hits across files, and ranks test files below the code they test unless the question asks about tests.

The benchmark harness changed too:

• The neuralmind backend runs read-only; it used to train the index it measured.
• Its tokens are counted with tiktoken like every other backend’s, instead of the selector’s chars/4 estimate.

The movement:

• requests recovers xfile-status-codes (0.96 → 1.00), click recovers echo-util (0.86 → 1.00), and flask recovers xfile-dispatch-context (0.95 → 1.00). rich stays at 1.00.
• Net: 3 misses → 0, found-rate 92.5% → 100%.
• Tokens per question fell on requests and click and rose on flask and rich, so the range moved to 51.2–242.0×. It was 45.3–245.2×.
• The embedding-rag baseline’s numbers moved with the new embedded text: fewer tokens per entry, and recall 0.85 on flask (was 0.95).

How each change was measured, and which ones were put back to check them: bench/retrieval/README.md.

Add a repo the same way: append to evals/public/manifest.json (pin the commit, give each query an objective def-site gold file) and re-run. Community-contributed real-repo numbers go through the existing neuralmind benchmark . --contribute path.

Competitor head-to-head — codebase-memory-mcp 0.8.1

The obvious incumbent (DeusData, single-binary, on-device embeddings, no LLM API key) runs headless, so we ran a real, reproducible row on the same pinned repos, same questions, same objective def-site gold, same quality.py scorer, retrieval depth matched to embedding-rag (top-8). Raw per-query traces: bench/public/competitor/. Reproduce: python -m evals.public.competitor.

At matched retrieval depth, NeuralMind finds the objectively-correct file every time and ranks it far higher (MRR 0.96 vs 0.23 on requests), while the competitor surfaces the gold file in its top-8 only ~half the time — and the files it does surface cost an order of magnitude more tokens to read.

Fairness & honest caveats (don’t skip)

• Most-favorable keyword mapping. The competitor takes a keyword array, not free text. We tested three reproducible mappings and used the one best for the competitor (all question words) — so this can’t be dismissed as crippling it.
• Pure retrieval on both sides, no agent loop. This measures retrieval ranking given a question — exactly how we test NeuralMind’s own search. The competitor’s published numbers (~90% of an “Explorer” agent across 31 scored languages; C at 0.58) come from an LLM-driven agent loop that isn’t reproducible without paying for an LLM; we don’t claim to reproduce that, and we cite their figures as-is. Their “158 languages” is vendored grammars, not all benchmarked.
• Cost is a proxy (tokens of the whole files the competitor surfaces at depth 8, since it returns paths the agent then reads — analogous to ripgrep).
• Full provenance + pinned version + exact commands: bench/public/competitor/REPRODUCE.md.