This development-only benchmark measures whether retrieval returns the banked turns needed to answer a coding question. It does not measure whether an agent completes the task, produces a correct answer, or saves time. A low score is a result, not a failing gate.
Dataset card
The checked-in corpus contains 432 bank entries across six fictional projects, with 216 English queries. Each category has 36 queries: 24 development and 12 test. The complete split is 144 development / 72 test queries. Original synthetic data and its authored sources carry the CC0-1.0 dedication in dataset/LICENSE; no third-party dataset was imported.
| Fictional project | Stack | Entries |
|---|---|---|
| Ledgerloom | Go, chi, pgx/sqlc, PostgreSQL | 72 |
| Tidewater | TypeScript, Expo/React Native, SQLite | 72 |
| Kestrel Ingest | Python, FastAPI, Celery/Redis, Polars/DuckDB, Parquet | 72 |
| Brindle | Rust, tokio, journald, compressed log shipping | 72 |
| Orchard Shop | Ruby/Rails, Hotwire, Sidekiq, MySQL | 72 |
| Quillpad | Elixir/Phoenix LiveView, Ecto, PostgreSQL, Oban | 72 |
Authorship. The corpus lead wrote Ledgerloom and Tidewater; its harness reported anthropic/claude-opus-5-5, not independently verified. Four task subagents wrote the other projects; their model identifiers were not exposed and are recorded as unknown, not guessed. The benchmark lead, openai-codex/gpt-6.1-sol, reviewed the generated sample. Full project descriptions and author provenance are in dataset/card.json.
How it was written. Stories and queries were authored as original coding turns, not extracted from real sessions. Commands, bugs, migrations, reversals and outcomes are fictional. A shared project template reserves six exact targets, six paraphrase targets, six old/new decision pairs, fifteen entries in six multi-entry answer sets, six cross-project scope topics, and twenty-seven background turns. Repeated structure is deliberate and limits generalization; the individual issues and fixes differ.
generate.ts derives banker-shaped titles, tags and capped action digests from these sources. Fixed seed 20261114 drives timestamps, opaque ids, a per-category one-third test split, and query ordering. It checks gold references, chronology, scope counterparts and category invariants; --check compares generated JSON byte-for-byte without writing it. Neither the generator nor the authors ran a retriever while authoring the corpus.
Query labels. Exact queries name identifiers or error details. Paraphrases use different content vocabulary across the gold title, tags, prompt, actions and result: lowercased word tokens with light suffix stripping may share only the generator's explicit function-word/generic-verb list (use, make, get, need, etc.). Superseded queries require the newer reversing decision. Multi-entry questions require complementary turns, not merely interchangeable matches. Scope queries have same-topic material in another project. Negatives ask an absent detail using plausible project vocabulary rather than only nonsense terms.
Lead review before measurement. A score-independent helper selected five entries per project at the first, quarter, midpoint, three-quarter and final positions in chronological project order, plus three development and two test queries per category with their full gold entries. The lead read all 30 entries and 30 queries. The check found no banker-cap violations or exact duplicate entry contents. Raw paraphrase overlaps outside a small linguistic stoplist were only function words, a generic verb, or an apostrophe fragment; none was distinctive. The corpus was then frozen before either lexical run; no question, label or text was changed in response to a score.
This represents authored coding-session retrieval questions only: not real developer distributions, personal-fact recall, non-English use, large production corpora, or downstream task success. The public test split holds out queries, not projects, authors or templates.
Reproduce
From the repository root, with the checkout's Bun dependencies installed:
bun packages/superpi/bench/memory/generate.ts
bun run --cwd packages/superpi/bench/memory check
(cd packages/superpi && bun test bench/memory/metrics.test.ts)
bun packages/superpi/bench/memory/run.ts --retriever lexical --json .worktrees/tmp/memory-baseline.jsonThe checked-in dataset is the benchmark input; generation is optional. Do not rewrite it in response to a retrieval score. Run every adapter on the identical corpus, scopes, query order and gold labels. An adapter receives entries and query text/scope, never gold labels or category/split metadata. One third of queries form the test split; report development and test results separately. The test labels are public, not a private evaluation service.
The runner indexes once and requests ten accepted hits per query. Search timing excludes corpus generation, process startup and indexing. It includes adapter search work, including result rendering. Fixed data and ordering make rankings reproducible; wall-clock latency is machine- and load-dependent. No warm-up searches or repeated-query best-of timing are used.
Production lexical baseline
adapters/lexical.ts uses LocalMemoryIndex and searchAllMemories from the production memory modules, with cloud search disabled. Bank entries are materialized as Markdown with frontmatter and ## Prompt, ## Actions, and ## Result sections. Database and memory roots are explicitly isolated; the adapter never indexes the user's memory tree.
The baseline therefore includes the actual bank parser, Porter-stemmed SQLite FTS5 index, meaningful-word OR-prefix query compilation, column-weighted BM25 ranking, production deduplication, and title/snippet rendering. It does not substitute a bag-of-words approximation. There is no additional fitted score threshold: today's local memory_search accepts FTS matches. The separate precision gate used by proactive context-hook suggestions is not the path being measured.
Metrics
Ranking metrics use binary relevance and macro-average answerable queries. Negatives are excluded from these means, rather than made perfect or zero-recall examples.
- Recall@1, @5, @10: distinct gold ids retrieved by the cutoff, divided by the number of distinct gold ids. Several-entry answers receive partial credit.
- MRR@10: reciprocal rank of the first gold hit within ten; zero on a miss.
- nDCG@10: sum of
1 / log2(rank + 1)for distinct relevant hits, divided by the ideal gain formin(number of gold ids, 10)relevant hits. - Negative false-positive rate: fraction of empty-gold queries returning any accepted hit. Each adapter applies its own production acceptance policy; the benchmark never invents a common numeric score cutoff.
- Duplicates in top five: mean count of repeated ids beyond their first occurrence per query. Duplicates consume ranking positions but never gain relevance credit twice. This is identity duplication, not a semantic-similarity judgment.
- Characters per hit: total returned
text.lengthdivided by total returned hit count, including duplicates. Units are JavaScript UTF-16 code units, including markup; not tokens, bytes, or Unicode code points. Full-entry fetches and outer MCP wrappers are excluded. - Search p50/p95: nearest-rank percentiles,
ceil(p × n), across one timed search per query.
Operational means include negatives. An inapplicable ranking metric or negative rate is null in JSON and n/a in tables. Empty operational aggregates are zero. Overall, split, and per-category tables must be read together: a strong exact-lookup average can hide paraphrase or temporal failures.
Extending retrieval
The interface in packages/superpi/bench/memory/retriever.ts is shared with retrieval implementers. Add adapters/<name>.ts with default export () => Promise<Retriever>; use run.ts --retriever <name>. index() is called once; search() returns best-first accepted hits; close() releases resources. projects: null means every project; an explicit list is the allowed project scope. Preserve opaque entry ids in results.
The runner selects an adapter through a temporary static-import bootstrap. This avoids dynamic imports while allowing a new adapter file without changing the runner. Adapters own score interpretation; the benchmark relies on ordering and accepted results, not comparable raw scores.
The child starts with a clean environment: its home, HarnessLink configuration, agent, XDG, and temporary directories all point inside a unique .worktrees/tmp/ directory; dotenv autoloading is disabled. The parent forwards termination signals and removes its scratch directory afterward. No account or running service is required.
JSON schema version 1 records the requested and actual retriever names, k, Bun version/platform/architecture, raw-file SHA-256 hashes and counts, project count, category/split counts, and metrics.{all,dev,test}.{overall,byCategory}. results records every query id/category/split, timed latency and ordered {id, score, text} hits. It contains no wall-clock timestamp; only timings vary on repeat runs.
Exit codes: 0 for a completed measurement regardless of quality, 1 for harness/data/adapter/I/O failure, and 2 for invalid arguments or an unknown adapter. --help lists available adapters.
Boundaries and limits
- Synthetic coding turns are not a sample of real developers, production repositories, or the founder's data. No private memory, session, transcript, secret, real person, or customer fixture is used.
- Authored gold labels reflect the author's intended question. They are not independent human consensus. Published labels and deterministic templates permit overfitting; a held-out query split is not an unseen distribution or independent project holdout.
- The corpus is small enough for a local index and does not characterize large-corpus scaling, cloud latency, cross-device fusion, linked-project discovery, or network failures.
- Temporal queries require the replacement decision, but retrieval alone does not prove an answer generator ignores obsolete evidence. Several-entry recall does not prove synthesis.
- Negative questions can share topic words with stored turns while requesting an absent detail. Returning a related but non-answering memory is a false positive under these labels.
- Payload size measures the snippet delivered before full retrieval, not answer completeness. Latency is a single local run, not a service-level objective.
The benchmark lives outside the CLI source graph. The package's release whitelist includes src, scripts, and CHANGELOG.md, not bench; no CLI command imports it. Only the arithmetic metric test file participates in the package's default bun test. Dataset generation and retrieval runs are explicit commands. Benchmark-local lint and type checks cover code omitted by the production compiler configuration.
Recorded lexical baseline
Measured with Bun 1.4.3, darwin-arm64, k=10. Production retrieval source: worktree base revision 85562553e5c82a290909d9ddd5a9a31eecbbf974, before the parallel memory redesign. Full per-query ordered hits and unrounded metrics: packages/superpi/bench/memory/results/lexical.json. Both runs exited 0; all 216 ordered results and non-timing metrics matched, with zero lingering runner scratch directories.
bun --no-env-file packages/superpi/bench/memory/run.ts --retriever lexical --json packages/superpi/bench/memory/results/lexical.json | tee .worktrees/tmp/lexical-baseline.log
bun --no-env-file packages/superpi/bench/memory/run.ts --retriever lexical --json .worktrees/tmp/lexical-repeat.json > .worktrees/tmp/lexical-repeat.logRaw input SHA-256:
- Entries:
a6399ed98f257a724935fc854476c23aed7c45e2a89d0bf4bdf8400288f56b48. - Queries:
847a4876fe93ee237a28ebfcac12a8408021733884145d601bbd3169b60f5ba3.
All queries
| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| overall | 216 | 180 | 36 | 0.521 | 0.761 | 0.806 | 0.696 | 0.719 | 1.000 | 0.000 | 241.0 | 0.65 | 1.12 |
| exact | 36 | 36 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 244.1 | 0.80 | 1.34 |
| paraphrase | 36 | 36 | 0 | 0.000 | 0.028 | 0.111 | 0.020 | 0.041 | n/a | 0.000 | 241.5 | 0.47 | 1.09 |
| superseded | 36 | 36 | 0 | 0.333 | 0.889 | 0.972 | 0.595 | 0.689 | n/a | 0.000 | 242.9 | 0.60 | 0.97 |
| multi | 36 | 36 | 0 | 0.384 | 0.917 | 0.972 | 0.948 | 0.933 | n/a | 0.000 | 235.8 | 0.68 | 1.15 |
| scope | 36 | 36 | 0 | 0.889 | 0.972 | 0.972 | 0.917 | 0.930 | n/a | 0.000 | 242.2 | 0.66 | 1.04 |
| negative | 36 | 0 | 36 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 239.7 | 0.63 | 1.22 |
Development split
| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| overall | 144 | 120 | 24 | 0.525 | 0.758 | 0.817 | 0.700 | 0.723 | 1.000 | 0.000 | 241.2 | 0.65 | 1.07 |
| exact | 24 | 24 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 246.9 | 0.80 | 1.06 |
| paraphrase | 24 | 24 | 0 | 0.000 | 0.000 | 0.125 | 0.016 | 0.040 | n/a | 0.000 | 240.2 | 0.50 | 0.96 |
| superseded | 24 | 24 | 0 | 0.333 | 0.917 | 1.000 | 0.595 | 0.696 | n/a | 0.000 | 241.4 | 0.60 | 0.72 |
| multi | 24 | 24 | 0 | 0.375 | 0.875 | 0.958 | 0.943 | 0.918 | n/a | 0.000 | 238.5 | 0.70 | 1.15 |
| scope | 24 | 24 | 0 | 0.917 | 1.000 | 1.000 | 0.948 | 0.961 | n/a | 0.000 | 241.6 | 0.66 | 0.94 |
| negative | 24 | 0 | 24 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 238.4 | 0.66 | 1.22 |
Test split
| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| overall | 72 | 60 | 12 | 0.514 | 0.767 | 0.783 | 0.687 | 0.710 | 1.000 | 0.000 | 240.7 | 0.64 | 1.20 |
| exact | 12 | 12 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 238.6 | 0.73 | 1.37 |
| paraphrase | 12 | 12 | 0 | 0.000 | 0.083 | 0.083 | 0.028 | 0.042 | n/a | 0.000 | 244.4 | 0.41 | 1.13 |
| superseded | 12 | 12 | 0 | 0.333 | 0.833 | 0.917 | 0.594 | 0.675 | n/a | 0.000 | 246.1 | 0.66 | 1.20 |
| multi | 12 | 12 | 0 | 0.403 | 1.000 | 1.000 | 0.958 | 0.962 | n/a | 0.000 | 230.2 | 0.67 | 0.96 |
| scope | 12 | 12 | 0 | 0.833 | 0.917 | 0.917 | 0.854 | 0.869 | n/a | 0.000 | 243.3 | 0.62 | 1.26 |
| negative | 12 | 0 | 12 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 242.7 | 0.62 | 0.91 |
Exact identifiers work well; vocabulary-disjoint paraphrases and selecting the current decision are much weaker. Every negative returned related material rather than an abstention. These are published limitations, not failures hidden by a score gate. Porter stemming, prefix matching, and allowed generic words can still retrieve a paraphrase gold entry despite the authoring vocabulary rule.