Benchmarks & Results

Everything here is measured and reproducible — no hand-picked or hardcoded numbers. Every figure is produced by code in the repo and gated in CI, so it can’t silently regress. Where a number is an estimate or a real-repo extrapolation, it says so. One labeled exception: the field report below is a one-repo, maintainer-measured case study — reproducible in method, not gated in CI.

Reproduce locally: python -m tests.benchmark.run (token reduction + learning

• synapse A/B), python -m evals.faithfulness.runner --run (answer quality), python -m evals.onboarding.runner --run (onboarding lift), python -m evals.parity.run (backend parity).

Four data-backed benefits (the short version)

NeuralMind is more than token reduction; the numbers below back four benefits. Two run on real, pinned OSS repos (requests, click, flask, rich) and are fully reproducible — python -m evals.public.run (methodology) — and two are committed A/Bs on the bundled reference fixture (real but smaller-scope): (1) Cheaper context — 79–100% gold-file recall (93.75% mean, 90% found-rate across 40 queries) at 45–257× fewer tokens than pasting files, beating ripgrep on both recall and cost on every repo; (2) Finds the right code — 100% gold-file recall, MRR 0.96, beating the incumbent codebase-memory-mcp on retrieval ranking (0.96 vs 0.23) — a separate, off-by-default eval on requests/click only, not yet re-verified against the current flask/rich-expanded corpus; (3) Learns how you work — the Hebbian synapse layer lifts top-k hit-rate, budget-neutral (reference fixture; +3.5 to +14 points across runs, CI gates the direction); (4) Better-grounded answers — at a matched budget its context carries more gold facts than naive truncation (reference fixture; delta +0.013 to +0.143 across runs, CI gates it at ≥ 0, grounding 1.00). We report where NeuralMind doesn’t win too — a well-tuned vector RAG ties or beats it on pure findability and is cheaper on raw tokens, two repos have partial gold-file misses (see the public benchmark’s “Where NeuralMind loses” section), and the competitor row is pure retrieval ranking, not their LLM-agent loop. Full tables and reproduction commands below.

The honest headline

On code questions, NeuralMind sends the agent the few entities that matter instead of whole files — so the same answer costs 12-50× fewer tokens on real repositories. That real-repo range is the product’s positioning; the number we measure in CI is deliberately conservative, on a tiny 500-line fixture where there’s little to prune, and it still clears a wide margin.

The fixture number is the floor of a floor: small repo, conservative gate. The mechanism is what scales — the bigger the codebase, the more whole-file context you avoid.

Does the memory make answers better, not just shorter?

Yes, and it’s measured. The faithfulness eval compares NeuralMind’s selected context against naive truncation at the same token budget — the honest comparison, not “small context vs the whole repo.”

A positive delta means smart selection beats dumb truncation at equal cost, and that is what CI guarantees. The size of the delta is not a fixed property: on a ~500-line fixture behind an HNSW index it moves between runs, so we publish the gate and the observed band rather than a point estimate that goes stale the week after it is written.

The learned memory layer (the differentiator)

NeuralMind’s moat is usage memory: a Hebbian synapse layer that learns what your team edits together and surfaces it on future queries. Both effects are measured by isolated A/Bs:

Both are budget-neutral by design: recalled nodes displace the weakest hits rather than adding tokens. The onboarding lift is the answer to “does an agent that inherits a committed team memory retrieve better on its first queries than a cold agent?” — gated in CI at lift ≥ 0.

v0.21.0 — ChromaDB-free retrieval, at parity

The opt-in turbovec backend (Google TurboQuant) can embed and search with zero ChromaDB, and it does so without giving up quality:

• The bundled embedder produces vectors byte-identical to ChromaDB’s (all-MiniLM-L6-v2): verified cosine 1.0, max elementwise diff 0.0 — so retrieval quality is unchanged; only the index representation differs.
• 8–16× smaller vectors means real memory headroom on large monorepos, and it retires the dependency behind the recurring CVE-2026-45829 advisory.

Multi-language & precision (structural parity, gated)

The built-in tree-sitter backend matches graphify symbol-for-symbol on the reference fixtures for all ten bundled languages (Python plus the nine above); an optional SCIP pass replaces heuristic call edges with compiler-accurate ones. All gated by evals/parity/run.py (coverage floor 90%, zero dangling edges) — the numbers above are emitted live by the parity gate on every PR. Per-language answer quality (vs structural coverage) is still Python-first; see Limits & Failure Modes.

Field report: a real-world rebuild (not CI-gated)

Unlike everything above, this is a field report: the maintainer ran NeuralMind across a major internal rebuild of a private, mid-size TypeScript SaaS platform (~9,300 nodes) and recorded before/after numbers with the shipped CLI (neuralmind stats / benchmark, a timed build --force). One repo, one developer, anonymized — reproducible in method on your own codebase, but not a CI-gated claim.

Full table, interpretation, and a step-by-step recipe for the same before/after measurement on your own refactor: Measure memory across a major refactor.

What we don’t claim

• The CI numbers come from a deliberately tiny fixture — they prove the mechanism and catch regressions, not a real-repo ceiling. Point it at your own repo with benchmark-your-repo.
• TurboQuant is an approximate (quantized) index; parity is gated on the reference fixture, and the compression win only matters at scale.
• The 12-50× figure is a real-repo range, not a fixed guarantee — your ratio depends on repo size and question shape.
• The field report above is a single private-repo measurement by the maintainer — treat it as an existence proof consistent with the 12-50× range, not an independent benchmark.

Reproduce every number

pip install -e ".[dev]" tiktoken
python -m tests.benchmark.run            # reduction + learning + synapse A/B
python -m evals.faithfulness.runner --run   # answer-quality delta
python -m evals.onboarding.runner --run     # onboarding lift
python -m evals.parity.run               # backend parity (incl. turbovec)

Each prints a report and exits non-zero if it falls below its gate — the same checks that run on every PR.