Deterministic — re-run in CI on every commit
Every number on this page comes from one of two sources: a deterministic public benchmark on pinned, real OSS repos that CI re-runs on every commit, and community submissions PR'd into a schema-validated dataset. No synthetic demos, no cherry-picking — every row ships with the exact command that reproduces it.
The public benchmark
A token-reduction number with no correctness number attached is meaningless. The public benchmark scores both at once: does the objectively-correct file land in the context window, and how many tokens did it cost to put it there? Built to be hard to dismiss:
Household-name OSS — requests, click, flask, rich — pinned to exact commit SHAs. Anyone can git checkout and audit; no vendor fixture.
Each query's gold file is the definition site of a named symbol, verifiable with one rg command. Deterministic — nothing to rig.
Not just naive file dumps: keyword search (ripgrep) and a vector RAG using the same encoder NeuralMind uses. Losses shown, not hidden.
Queries are committed in evals/public/manifest.json before tuning. Every one is reported; re-running yields identical numbers.
requests @ 0e322af877| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 41,729 | 1× |
| ripgrep | 0.79 | 26,543 | 1.6× |
| embedding-rag | 1.00 | 607 | 69× |
| neuralmind | 0.96 | 930 | 44.9× |
click @ 874ca2bc1c| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 78,514 | 1× |
| ripgrep | 0.79 | 45,059 | 1.7× |
| embedding-rag | 1.00 | 634 | 123.8× |
| neuralmind | 0.79 | 788 | 99.6× |
flask @ c12a5d874c| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 59,013 | 1× |
| ripgrep | 0.85 | 26,891 | 2.2× |
| embedding-rag | 0.95 | 687 | 85.9× |
| neuralmind | 0.95 | 772 | 76.4× |
rich @ 7f580bdc70| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 232,483 | 1× |
| ripgrep | 1.00 | 43,437 | 5.4× |
| embedding-rag | 1.00 | 677 | 343.2× |
| neuralmind | 1.00 | 905 | 256.8× |
Read the losses too: across all 4 repos (40 queries), NeuralMind's weighted mean recall is 93.75% (90% found-rate) — click is the weakest repo at 0.79 recall (2 misses of 7); requests and flask each miss 1 query. Where embedding-rag also hits full or near-full recall, it's NeuralMind's own vector-retrieval core in isolation, and it's cheaper on raw tokens — the gap is what the progressive-disclosure assembly layer (project map + symbols + call edges) adds on top of raw top-k chunks. Scoring reuses neuralmind/quality.py verbatim — the same metric code the CI quality gate runs. Full method, baselines, and raw per-query data in the methodology.
# Reproduce it yourself — no trust required. Deterministic: # your numbers match the tables above to the token. git clone https://github.com/dfrostar/neuralmind && cd neuralmind pip install -e . tiktoken python -m evals.public.run
Community numbers
Each entry below was produced by running neuralmind benchmark . --json on the submitter's own repository and PR'd into the dataset. CI validates every submission against a schema. NeuralMind never uploads anything automatically — these are deliberate, auditable contributions.
Each point is one submission — graph node count (a proxy for codebase size) against the average reduction ratio vs loading the whole repo. Hover any point for the project, language, and exact numbers.
Average reduction ratio per language, with the submission count under each bar — so you can tell "this is solid" apart from "n=1, take with salt".
The dataset behind both charts. The verification command for every row is neuralmind benchmark . --json — run it on your own repo and the numbers should reproduce in the same range.
| Project | Language | Nodes | Reduction | Wakeup tok | Query tok | Model | Submitted by | Date | Notes |
|---|---|---|---|---|---|---|---|---|---|
| Loading submissions… | |||||||||
Click a column header to sort. Data loads from community-benchmarks.json at page load — no server, no tracking.
Contribute
The dataset gets stronger with every repo it covers — especially across different languages and sizes. Five minutes, two commands, one PR:
pip install neuralmind cd /path/to/your-repo neuralmind build . # prints your numbers + a ready-to-share JSON blob neuralmind benchmark . --contribute
Then open a PR adding the blob as one entry in docs/community-benchmarks.json (fields enforced by community-benchmarks.schema.json), or file it via the benchmark submission issue template. CI validates the entry; this dashboard updates automatically once it merges.