Benchmarks & Results

Everything here is measured and reproducible — no hand-picked or hardcoded numbers. Every figure is produced by code in the repo. The fixture-based figures are gated in CI, so they can’t silently regress; the public benchmark on real OSS repos and the multi-repo retrieval eval are reproducible on demand (deterministic, one command, raw data committed) but are not CI gates. Where a number is an estimate or a real-repo extrapolation, it says so. One labeled exception: the field report below is a one-repo, maintainer-measured case study — reproducible in method, not gated in CI. On the public benchmark, “reproducible” means gold-file recall and found-rate have come back identical on every machine compared, while for the current run per-repo mean tokens differ by up to 0.8% between the GitHub-hosted CI runners with and without AVX-512, and so does the vector-RAG baseline’s MRR on click (an Apple M3, also without AVX-512, matched the committed run exactly) (details).

Reproduce locally: python -m tests.benchmark.run (token reduction + learning

• synapse A/B), python -m evals.faithfulness.runner --run (answer quality), python -m evals.onboarding.runner --run (onboarding lift), python -m evals.parity.run (backend parity), NEURALMIND_ORT_THREADS=1 python -m evals.retrieval.run --public-benchmark (multi-repo retrieval eval; needs pip install -e . tiktoken).

What the data shows, losses included (the short version)

NeuralMind is more than token reduction; the numbers below cover four benefits. Two run on real, pinned OSS repos (requests, click, flask, rich) and are fully reproducible — python -m evals.public.run (methodology) — and two are committed A/Bs on the bundled reference fixture (real but smaller-scope): (1) Cheaper context — the gold file on 40 of 40 queries at 51–242× fewer tokens than pasting files, beating ripgrep on cost on every repo and on recall on three of four (tying on the fourth); (2) Finds the right code — 100% gold-file recall, MRR 0.96, beating the incumbent codebase-memory-mcp on retrieval ranking (0.96 vs 0.23) — a separate, off-by-default eval on requests/click only, not yet re-verified against the current flask/rich-expanded corpus; (3) Learns how you work — the Hebbian synapse layer never lowers top-k hit-rate, budget-neutral (reference fixture; +3.5 to +14 points in earlier releases, +0 at v4.12.0 where the fixture is at 97% without it; CI gates the direction); (4) Answer grounding vs. naive truncation — at a matched budget on the prose-heavy reference fixture, delta +0.265 at v4.12.0 (measured with CI’s command on one machine; v4.11.1 +0.027 there), after a published loss of −0.054 at v4.3.4; CI fails below −0.10. We report where NeuralMind doesn’t win too — the plain vector baseline (the top 8 entries of NeuralMind’s own index: names and docstrings, not code) is cheaper on raw tokens on every repo and still ranks the gold file higher on flask and rich (MRR 0.74 vs 0.72, 0.94 vs 0.83), though its recall is at or below NeuralMind’s; 40 queries is a small sample, and the 150-question retrieval eval misses 10 (see the public benchmark’s “Where NeuralMind loses” section), and the competitor row is pure retrieval ranking, not their LLM-agent loop. Full tables and reproduction commands below.

The honest headline

On code questions, NeuralMind sends the agent the few entities that matter instead of whole files — so the same answer costs 12-50× fewer tokens on real repositories. That real-repo range is the product’s positioning; the number we measure in CI is deliberately conservative, on a tiny 500-line fixture where there’s little to prune, and it still clears a wide margin.

The fixture number is the floor of a floor: small repo, conservative gate. The mechanism is what scales — the bigger the codebase, the more whole-file context you avoid.

Does the memory make answers better, not just shorter?

Not on this measure, right now — and we publish that. The faithfulness eval compares NeuralMind’s selected context against naive truncation at the same token budget — the honest comparison, not “small context vs the whole repo.”

A positive delta would mean smart selection beats plain truncation at equal cost. At v4.3.4 it doesn’t: the reference fixture mixes code with prose chapter summaries, and on prose a matched-budget truncation can keep more of the expected facts — which is why ci-benchmark.yml gates at −0.10 rather than 0. The gate catches a real regression; it does not guarantee a win. The size of the delta moves with retrieval changes, so read the current value from the CI benchmark comment on any pull request.

The learned memory layer (the differentiator)

NeuralMind’s moat is usage memory: a Hebbian synapse layer that learns what your team edits together and surfaces it on future queries. Both effects are measured by isolated A/Bs:

Both are budget-neutral by design: recalled nodes displace the weakest hits rather than adding tokens. The onboarding lift is the answer to “does an agent that inherits a committed team memory retrieve better on its first queries than a cold agent?” — gated in CI at lift ≥ 0.

Retrieval eval (v4.12.0)

Evidence level: reproducible on demand — raw output committed in bench/retrieval/vs-v4.11 and bench/retrieval/on-v4.12; not a CI gate.

v4.12.0 against v4.11.1, paired question by question (--compare), on 150 questions over five public repositories (requests, click, flask, rich, this repository):

On the 40-query public benchmark, the gold file is now found on all 40 queries (v4.11.1: 95% recall, 85.71–100% per repo), and the ratio went from 45–246× to 51–242× fewer tokens than pasting every file.

21 questions won and 1 lost on hit@5 (exact McNemar p < 0.0001); every repository improved (requests +3, click +4, flask +3, rich +3, this repository +7); the mean MRR change is +0.079, paired bootstrap 95% interval [+0.024, +0.132]. Each default was then put back one at a time (on-v4.12): four L3 hits instead of eight costs 4 questions, re-weighting by detected intent 4, the code-signal boost 1, turning off the spread across files 1, and ranking tests like code 1 (and 0.020 MRR). Since v4.12.0 the keep rule is paired: a configuration is kept when, pooled over every question, it wins more hit@5 questions than it loses with an exact McNemar p < 0.05, no repository drops by more than two questions, and tokens rise by at most 10%. The v4.6.0 rounds below used the rule this replaced.

Losses: 10 of the 150 questions are still missed, six of them on this repository; on flask and rich NeuralMind’s MRR stays below the plain vector baseline’s. Answer quality with a model in the loop is not measured.

Retrieval eval (v4.6.0)

Evidence level: reproducible on demand — one command, raw output committed in bench/retrieval/; not a CI gate.

v4.6.0 asked which ranking changes earn a default, and answered it with a multi-repo eval instead of one repo’s anecdote. python -m evals.retrieval.run asks 30 questions per repository — pre-registered and committed in evals/retrieval/questions/ (before any ranking change was tried) for the five public repositories, plus a private 383-file repository with its own local question set that readers can’t re-run — under every flag configuration, read-only. The repositories: requests, click, flask and rich at the public benchmark’s pinned commits, this repository (with its docs indexed), and a private 383-file repository, reported only in aggregate.

The keep rule, fixed before the runs:

• mean hit@5 across repos goes up, and it goes up on at least 3 repos;
• no repo drops by more than one question;
• average context tokens rise by at most 10%;
• the public 4-repo benchmark’s gold-file recall doesn’t drop.

What was kept: one BM25 index over docs and code. The default turbovec backend’s keyword index held only document nodes, so the hybrid fusion gave docs a keyword signal code never got. v4.6.0 indexes every node (doc text, symbol names, file paths, docstrings) in one BM25 index, on by default (NEURALMIND_BM25_UNIFIED=0 restores v4.5.0). Against v4.5.0 (bench/retrieval/vs-v4.5), hit@5 / MRR:

On the five public repositories alone, mean hit@5 / MRR goes from 75.3% / 0.614 to 80.7% / 0.664. Context tokens rose 1.1%, and the public benchmark’s gold-file recall went from 93.75% to 95%. The losses, published: requests drops one question (90% → 87%), rich’s MRR falls from 0.71 to 0.60, and on the public benchmark click gains a miss (echo-util), falling from 100% to 85.71%. The private 383-file repository’s 60% baseline is the figure the work started from; its target (hit@5 ≥ 80%, MRR ≥ 0.65) is not met — it reaches 73% / 0.60.

What was measured and not kept. Each stays available behind its flag, off by default:

One finding worth keeping even though its fix wasn’t: the old intent classifier (still the default, since the intent rules weren’t kept) sends many questions about how the project behaves — “how does X…”, “where is X…”, “which X is…” — to docs intent, which multiplies doc hits ×2.0 and code ×0.7. Four questions of those shapes are now regression tests (tests/test_l3_slots_v460.py). Run neuralmind query --explain to see which intent L3 ranked a query with, and why.

pip install -e . tiktoken
NEURALMIND_ORT_THREADS=1 python -m evals.retrieval.run --public-benchmark --out bench/retrieval/on-v4.6
python -m evals.retrieval.run --private ~/work/your-repo   # add your own repo, locally

v0.21.0 — ChromaDB-free retrieval, at parity

The opt-in turbovec backend (Google TurboQuant) can embed and search with zero ChromaDB, and it does so without giving up quality:

• The bundled embedder produces vectors byte-identical to ChromaDB’s (all-MiniLM-L6-v2): verified cosine 1.0, max elementwise diff 0.0 — so retrieval quality is unchanged; only the index representation differs.
• 8–16× smaller vectors means real memory headroom on large monorepos, and it retires the dependency behind the recurring CVE-2026-45829 advisory.

Multi-language & precision (structural parity, gated)

The built-in tree-sitter backend matches graphify symbol-for-symbol on the reference fixtures for all ten bundled languages (Python plus the nine above); an optional SCIP pass replaces heuristic call edges with compiler-accurate ones. All gated by evals/parity/run.py (coverage floor 90%, zero dangling edges) — the numbers above are emitted live by the parity gate on every PR. Per-language answer quality (vs structural coverage) is still Python-first; see Limits & Failure Modes.

Field report: a real-world rebuild (not CI-gated)

Unlike everything above, this is a field report: the maintainer ran NeuralMind across a major internal rebuild of a private, mid-size TypeScript SaaS platform (~9,300 nodes) and recorded before/after numbers with the shipped CLI (neuralmind stats / benchmark, a timed build --force). One repo, one developer, anonymized — reproducible in method on your own codebase, but not a CI-gated claim.

Full table, interpretation, and a step-by-step recipe for the same before/after measurement on your own refactor: Measure memory across a major refactor.

Tool-output compression (measured, withdrawn, and one opt-in)

Through v4.4.0, install-hooks registered PostToolUse hooks that handed Claude compressed copies of Read, Bash and Grep output. The compression benchmark drives the real hook with Claude Code-shaped payloads and applies each response the way Claude Code’s documented hook protocol does.

Claude Code adds a hook’s additionalContext next to the tool result rather than replacing it, so the copies cost tokens. From v4.5.0 the hooks inject nothing by default. The compressors themselves would cut 70–87% if they replaced a result, but they keep 0% of a file’s source lines and 0% of a diff’s changed lines.

The one replacement that ships is opt-in (NEURALMIND_BASH_REPLACE=1, v4.10.0+) and narrow: it removes the progress lines of pip install and neuralmind build output through updatedToolOutput, and nothing else.

Per noisy-log call: mean −54.8%, from 0% (next build, not on the allowlist) to −96.5%. Every must-keep line of the replaced calls reaches Claude. CI recomputes the Bash results on every PR and fails if a replaced call keeps under 95% of its must-keep lines, if a content output, Read or Grep result is replaced, or if any call costs more tokens than with no hook (tests/test_compression_benchmark.py).

What we don’t claim

• The CI numbers come from a deliberately tiny fixture — they prove the mechanism and catch regressions, not a real-repo ceiling. Point it at your own repo with benchmark-your-repo.
• TurboQuant is an approximate (quantized) index; parity is gated on the reference fixture, and the compression win only matters at scale.
• The 12-50× figure is a real-repo range, not a fixed guarantee — your ratio depends on repo size and question shape.
• The field report above is a single private-repo measurement by the maintainer — treat it as an existence proof consistent with the 12-50× range, not an independent benchmark.

Reproduce every number

pip install -e ".[dev]" tiktoken
python -m tests.benchmark.run            # reduction + learning + synapse A/B
python -m evals.faithfulness.runner --run   # answer-quality delta
python -m evals.onboarding.runner --run     # onboarding lift
python -m evals.parity.run               # backend parity (incl. turbovec)

Each prints a report and exits non-zero if it falls below its gate — the same checks that run on every PR.