Deterministic — re-run in CI on every commit
Every number on this page comes from one of two sources: a deterministic public benchmark on pinned, real OSS repos that CI re-runs on every commit, and community submissions PR'd into a schema-validated dataset. No synthetic demos, no cherry-picking — every row ships with the exact command that reproduces it.
The public benchmark
A token-reduction number with no correctness number attached is meaningless. The public benchmark scores both at once: does the objectively-correct file land in the context window, and how many tokens did it cost to put it there? Built to be hard to dismiss:
Household-name OSS — requests, click, flask, rich — pinned to exact commit SHAs. Anyone can git checkout and audit; no vendor fixture.
Each query's gold file is the definition site of a named symbol, verifiable with one rg command. Deterministic — nothing to rig.
Not just naive file dumps: keyword search (ripgrep) and a vector baseline that returns the top 8 entries of NeuralMind’s own vector index — a symbol’s name, module and docstring, not its code, so its token count is a floor. Losses shown, not hidden.
Queries are committed in evals/public/manifest.json before tuning. Every one is reported. Re-runs have matched recall and found-rate exactly on every machine compared; token counts, and on one repo the vector-RAG baseline's MRR, can differ slightly between machines (how much).
requests @ 0e322af877| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 41,729 | 1× |
| ripgrep | 0.79 | 26,543 | 1.6× |
| embedding-rag | 1.00 | 229 | 181.9× |
| neuralmind | 1.00 | 816 | 51.2× |
click @ 874ca2bc1c| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 78,514 | 1× |
| ripgrep | 0.79 | 45,059 | 1.7× |
| embedding-rag | 1.00 | 220 | 357.5× |
| neuralmind | 1.00 | 670 | 117.1× |
flask @ c12a5d874c| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 59,013 | 1× |
| ripgrep | 0.85 | 26,891 | 2.2× |
| embedding-rag | 0.85 | 272 | 217.0× |
| neuralmind | 1.00 | 861 | 68.5× |
rich @ 7f580bdcf0| Backend | Gold-file recall | Mean tokens/query | vs full-file |
|---|---|---|---|
| full-file | 1.00 | 232,483 | 1× |
| ripgrep | 1.00 | 43,437 | 5.4× |
| embedding-rag | 1.00 | 236 | 985.5× |
| neuralmind | 1.00 | 961 | 242.0× |
Read the losses too: NeuralMind finds the gold file on all 40 queries, but 40 is a small sample: on the 150-question retrieval eval it misses 10, listed in bench/retrieval. embedding-rag is the top 8 entries of NeuralMind's own vector index, nothing added. NeuralMind's recall is at or above it on every repo, but it is cheaper on all four, and it still ranks the gold file higher on flask (MRR 0.74 vs 0.72) and rich (0.94 vs 0.83). Its token count is a floor: it sends a symbol's name, module and docstring, not code, where NeuralMind spends the difference on a structured context (project map, the query's other candidates, eight ranked hits). Scoring reuses neuralmind/quality.py verbatim — the same metric code the CI quality gate runs. Full method, baselines, and raw per-query data in the methodology. These tables were regenerated at v4.12.0, with the neuralmind backend run read-only and its tokens counted with the benchmark's tokenizer — what changed.
# Reproduce it yourself — no trust required. Recall and found-rate should match # the tables above exactly; token counts can differ slightly by machine: the # GitHub-hosted CI runners without AVX-512 land up to 0.8% off on a per-repo mean. git clone https://github.com/dfrostar/neuralmind && cd neuralmind pip install -e . tiktoken NEURALMIND_ORT_THREADS=1 python -m evals.public.run
Community numbers
Each entry below was produced by running neuralmind benchmark . --json on the submitter's own repository and PR'd into the dataset. CI validates every submission against a schema. NeuralMind never uploads anything automatically — these are deliberate, auditable contributions.
Each point is one submission — graph node count (a proxy for codebase size) against the average reduction ratio vs the CLI's fixed 50,000-token estimate, the one ratio every row shares. That ratio is 50,000 ÷ tokens per question, so this chart shows how context size moves with repo size, not the ratio against each repo's own code: v4.5.0+ submissions add that, in the table below. Hover any point for the project, language, and exact numbers.
Average reduction ratio per language, with the submission count under each bar — so you can tell "this is solid" apart from "n=1, take with salt".
The dataset behind both charts. On v4.5.0+, neuralmind benchmark . --naive-50k --json reproduces a row's fixed-estimate ratio and neuralmind benchmark . --json its ratio against the measured code — run them on your own repo and the numbers should reproduce in the same range.
| Project | Language | Nodes | vs 50K est. | vs measured code | Wakeup tok | Query tok | Model | Submitted by | Date | Notes |
|---|---|---|---|---|---|---|---|---|---|---|
| Loading submissions… | ||||||||||
Click a column header to sort. Data loads from community-benchmarks.json at page load — no server, no tracking.
Contribute
The dataset gets stronger with every repo it covers — especially across different languages and sizes. Five minutes, two commands, one PR:
pip install neuralmind cd /path/to/your-repo neuralmind build . # prints your numbers + a ready-to-share JSON blob neuralmind benchmark . --contribute
Then open a PR adding the blob as one entry in docs/community-benchmarks.json (fields enforced by community-benchmarks.schema.json), or file it via the benchmark submission issue template. CI validates the entry; this dashboard updates automatically once it merges.