# Coding-memory retrieval benchmark

This development-only benchmark measures whether retrieval returns the banked turns needed to answer a coding question. It does not measure whether an agent completes the task, produces a correct answer, or saves time. A low score is a result, not a failing gate.

## Dataset card

The checked-in corpus contains 432 bank entries across six fictional projects, with 216 English queries. Each category has 36 queries: 24 development and 12 test. The complete split is 144 development / 72 test queries. Original synthetic data and its authored sources carry the CC0-1.0 dedication in `dataset/LICENSE`; no third-party dataset was imported.

| Fictional project | Stack | Entries |
| --- | --- | --- |
| Ledgerloom | Go, chi, pgx/sqlc, PostgreSQL | 72 |
| Tidewater | TypeScript, Expo/React Native, SQLite | 72 |
| Kestrel Ingest | Python, FastAPI, Celery/Redis, Polars/DuckDB, Parquet | 72 |
| Brindle | Rust, tokio, journald, compressed log shipping | 72 |
| Orchard Shop | Ruby/Rails, Hotwire, Sidekiq, MySQL | 72 |
| Quillpad | Elixir/Phoenix LiveView, Ecto, PostgreSQL, Oban | 72 |

**Authorship.** The corpus lead wrote Ledgerloom and Tidewater; its harness reported `anthropic/claude-opus-5-5`, not independently verified. Four task subagents wrote the other projects; their model identifiers were not exposed and are recorded as unknown, not guessed. The benchmark lead, `openai-codex/gpt-6.1-sol`, reviewed the generated sample. Full project descriptions and author provenance are in `dataset/card.json`.

**How it was written.** Stories and queries were authored as original coding turns, not extracted from real sessions. Commands, bugs, migrations, reversals and outcomes are fictional. A shared project template reserves six exact targets, six paraphrase targets, six old/new decision pairs, fifteen entries in six multi-entry answer sets, six cross-project scope topics, and twenty-seven background turns. Repeated structure is deliberate and limits generalization; the individual issues and fixes differ.

`generate.ts` derives banker-shaped titles, tags and capped action digests from these sources. Fixed seed **20261114** drives timestamps, opaque ids, a per-category one-third test split, and query ordering. It checks gold references, chronology, scope counterparts and category invariants; `--check` compares generated JSON byte-for-byte without writing it. Neither the generator nor the authors ran a retriever while authoring the corpus.

**Query labels.** Exact queries name identifiers or error details. Paraphrases use different content vocabulary across the gold title, tags, prompt, actions and result: lowercased word tokens with light suffix stripping may share only the generator's explicit function-word/generic-verb list (`use`, `make`, `get`, `need`, etc.). Superseded queries require the newer reversing decision. Multi-entry questions require complementary turns, not merely interchangeable matches. Scope queries have same-topic material in another project. Negatives ask an absent detail using plausible project vocabulary rather than only nonsense terms.

**Lead review before measurement.** A score-independent helper selected five entries per project at the first, quarter, midpoint, three-quarter and final positions in chronological project order, plus three development and two test queries per category with their full gold entries. The lead read all 30 entries and 30 queries. The check found no banker-cap violations or exact duplicate entry contents. Raw paraphrase overlaps outside a small linguistic stoplist were only function words, a generic verb, or an apostrophe fragment; none was distinctive. The corpus was then frozen before either lexical run; no question, label or text was changed in response to a score.

This represents authored coding-session retrieval questions only: not real developer distributions, personal-fact recall, non-English use, large production corpora, or downstream task success. The public test split holds out queries, not projects, authors or templates.

## Reproduce

From the repository root, with the checkout's Bun dependencies installed:

```sh
bun packages/superpi/bench/memory/generate.ts
bun run --cwd packages/superpi/bench/memory check
(cd packages/superpi && bun test bench/memory/metrics.test.ts)
bun packages/superpi/bench/memory/run.ts --retriever lexical --json .worktrees/tmp/memory-baseline.json
```

The checked-in dataset is the benchmark input; generation is optional. Do not rewrite it in response to a retrieval score. Run every adapter on the identical corpus, scopes, query order and gold labels. An adapter receives entries and query text/scope, never gold labels or category/split metadata. One third of queries form the test split; report development and test results separately. The test labels are public, not a private evaluation service.

The runner indexes once and requests ten accepted hits per query. Search timing excludes corpus generation, process startup and indexing. It includes adapter search work, including result rendering. Fixed data and ordering make rankings reproducible; wall-clock latency is machine- and load-dependent. No warm-up searches or repeated-query best-of timing are used.

## Production lexical baseline

`adapters/lexical.ts` uses `LocalMemoryIndex` and `searchAllMemories` from the production memory modules, with cloud search disabled. Bank entries are materialized as Markdown with frontmatter and `## Prompt`, `## Actions`, and `## Result` sections. Database and memory roots are explicitly isolated; the adapter never indexes the user's memory tree.

The baseline therefore includes the actual bank parser, Porter-stemmed SQLite FTS5 index, meaningful-word OR-prefix query compilation, column-weighted BM25 ranking, production deduplication, and title/snippet rendering. It does not substitute a bag-of-words approximation. There is no additional fitted score threshold: today's local `memory_search` accepts FTS matches. The separate precision gate used by proactive context-hook suggestions is not the path being measured.

## Metrics

Ranking metrics use binary relevance and macro-average **answerable** queries. Negatives are excluded from these means, rather than made perfect or zero-recall examples.

- **Recall@1, @5, @10:** distinct gold ids retrieved by the cutoff, divided by the number of distinct gold ids. Several-entry answers receive partial credit.
- **MRR@10:** reciprocal rank of the first gold hit within ten; zero on a miss.
- **nDCG@10:** sum of `1 / log2(rank + 1)` for distinct relevant hits, divided by the ideal gain for `min(number of gold ids, 10)` relevant hits.
- **Negative false-positive rate:** fraction of empty-gold queries returning any accepted hit. Each adapter applies its own production acceptance policy; the benchmark never invents a common numeric score cutoff.
- **Duplicates in top five:** mean count of repeated ids beyond their first occurrence per query. Duplicates consume ranking positions but never gain relevance credit twice. This is identity duplication, not a semantic-similarity judgment.
- **Characters per hit:** total returned `text.length` divided by total returned hit count, including duplicates. Units are JavaScript UTF-16 code units, including markup; not tokens, bytes, or Unicode code points. Full-entry fetches and outer MCP wrappers are excluded.
- **Search p50/p95:** nearest-rank percentiles, `ceil(p × n)`, across one timed search per query.

Operational means include negatives. An inapplicable ranking metric or negative rate is `null` in JSON and `n/a` in tables. Empty operational aggregates are zero. Overall, split, and per-category tables must be read together: a strong exact-lookup average can hide paraphrase or temporal failures.

## Extending retrieval

The interface in `packages/superpi/bench/memory/retriever.ts` is shared with retrieval implementers. Add `adapters/<name>.ts` with default export `() => Promise<Retriever>`; use `run.ts --retriever <name>`. `index()` is called once; `search()` returns best-first accepted hits; `close()` releases resources. `projects: null` means every project; an explicit list is the allowed project scope. Preserve opaque entry ids in results.

The runner selects an adapter through a temporary static-import bootstrap. This avoids dynamic imports while allowing a new adapter file without changing the runner. Adapters own score interpretation; the benchmark relies on ordering and accepted results, not comparable raw scores.

The child starts with a clean environment: its home, HarnessLink configuration, agent, XDG, and temporary directories all point inside a unique `.worktrees/tmp/` directory; dotenv autoloading is disabled. The parent forwards termination signals and removes its scratch directory afterward. No account or running service is required.

JSON schema version 1 records the requested and actual retriever names, `k`, Bun version/platform/architecture, raw-file SHA-256 hashes and counts, project count, category/split counts, and `metrics.{all,dev,test}.{overall,byCategory}`. `results` records every query id/category/split, timed latency and ordered `{id, score, text}` hits. It contains no wall-clock timestamp; only timings vary on repeat runs.

Exit codes: 0 for a completed measurement regardless of quality, 1 for harness/data/adapter/I/O failure, and 2 for invalid arguments or an unknown adapter. `--help` lists available adapters.

## Boundaries and limits

- Synthetic coding turns are not a sample of real developers, production repositories, or the founder's data. No private memory, session, transcript, secret, real person, or customer fixture is used.
- Authored gold labels reflect the author's intended question. They are not independent human consensus. Published labels and deterministic templates permit overfitting; a held-out query split is not an unseen distribution or independent project holdout.
- The corpus is small enough for a local index and does not characterize large-corpus scaling, cloud latency, cross-device fusion, linked-project discovery, or network failures.
- Temporal queries require the replacement decision, but retrieval alone does not prove an answer generator ignores obsolete evidence. Several-entry recall does not prove synthesis.
- Negative questions can share topic words with stored turns while requesting an absent detail. Returning a related but non-answering memory is a false positive under these labels.
- Payload size measures the snippet delivered before full retrieval, not answer completeness. Latency is a single local run, not a service-level objective.

The benchmark lives outside the CLI source graph. The package's release whitelist includes `src`, `scripts`, and `CHANGELOG.md`, not `bench`; no CLI command imports it. Only the arithmetic metric test file participates in the package's default `bun test`. Dataset generation and retrieval runs are explicit commands. Benchmark-local lint and type checks cover code omitted by the production compiler configuration.

## Recorded lexical baseline

Measured with Bun 1.4.3, darwin-arm64, k=10. Production retrieval source: worktree base revision `85562553e5c82a290909d9ddd5a9a31eecbbf974`, before the parallel memory redesign. Full per-query ordered hits and unrounded metrics: `packages/superpi/bench/memory/results/lexical.json`. Both runs exited 0; all 216 ordered results and non-timing metrics matched, with zero lingering runner scratch directories.

```sh
bun --no-env-file packages/superpi/bench/memory/run.ts --retriever lexical --json packages/superpi/bench/memory/results/lexical.json | tee .worktrees/tmp/lexical-baseline.log
bun --no-env-file packages/superpi/bench/memory/run.ts --retriever lexical --json .worktrees/tmp/lexical-repeat.json > .worktrees/tmp/lexical-repeat.log
```

Raw input SHA-256:

- Entries: `a6399ed98f257a724935fc854476c23aed7c45e2a89d0bf4bdf8400288f56b48`.
- Queries: `847a4876fe93ee237a28ebfcac12a8408021733884145d601bbd3169b60f5ba3`.

### All queries

| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| overall | 216 | 180 | 36 | 0.521 | 0.761 | 0.806 | 0.696 | 0.719 | 1.000 | 0.000 | 241.0 | 0.65 | 1.12 |
| exact | 36 | 36 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 244.1 | 0.80 | 1.34 |
| paraphrase | 36 | 36 | 0 | 0.000 | 0.028 | 0.111 | 0.020 | 0.041 | n/a | 0.000 | 241.5 | 0.47 | 1.09 |
| superseded | 36 | 36 | 0 | 0.333 | 0.889 | 0.972 | 0.595 | 0.689 | n/a | 0.000 | 242.9 | 0.60 | 0.97 |
| multi | 36 | 36 | 0 | 0.384 | 0.917 | 0.972 | 0.948 | 0.933 | n/a | 0.000 | 235.8 | 0.68 | 1.15 |
| scope | 36 | 36 | 0 | 0.889 | 0.972 | 0.972 | 0.917 | 0.930 | n/a | 0.000 | 242.2 | 0.66 | 1.04 |
| negative | 36 | 0 | 36 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 239.7 | 0.63 | 1.22 |

### Development split

| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| overall | 144 | 120 | 24 | 0.525 | 0.758 | 0.817 | 0.700 | 0.723 | 1.000 | 0.000 | 241.2 | 0.65 | 1.07 |
| exact | 24 | 24 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 246.9 | 0.80 | 1.06 |
| paraphrase | 24 | 24 | 0 | 0.000 | 0.000 | 0.125 | 0.016 | 0.040 | n/a | 0.000 | 240.2 | 0.50 | 0.96 |
| superseded | 24 | 24 | 0 | 0.333 | 0.917 | 1.000 | 0.595 | 0.696 | n/a | 0.000 | 241.4 | 0.60 | 0.72 |
| multi | 24 | 24 | 0 | 0.375 | 0.875 | 0.958 | 0.943 | 0.918 | n/a | 0.000 | 238.5 | 0.70 | 1.15 |
| scope | 24 | 24 | 0 | 0.917 | 1.000 | 1.000 | 0.948 | 0.961 | n/a | 0.000 | 241.6 | 0.66 | 0.94 |
| negative | 24 | 0 | 24 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 238.4 | 0.66 | 1.22 |

### Test split

| Segment | N | Answerable | Negatives | R@1 | R@5 | R@10 | MRR@10 | nDCG@10 | Neg-FP | Dup@5 | Chars/hit | p50 ms | p95 ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| overall | 72 | 60 | 12 | 0.514 | 0.767 | 0.783 | 0.687 | 0.710 | 1.000 | 0.000 | 240.7 | 0.64 | 1.20 |
| exact | 12 | 12 | 0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | n/a | 0.000 | 238.6 | 0.73 | 1.37 |
| paraphrase | 12 | 12 | 0 | 0.000 | 0.083 | 0.083 | 0.028 | 0.042 | n/a | 0.000 | 244.4 | 0.41 | 1.13 |
| superseded | 12 | 12 | 0 | 0.333 | 0.833 | 0.917 | 0.594 | 0.675 | n/a | 0.000 | 246.1 | 0.66 | 1.20 |
| multi | 12 | 12 | 0 | 0.403 | 1.000 | 1.000 | 0.958 | 0.962 | n/a | 0.000 | 230.2 | 0.67 | 0.96 |
| scope | 12 | 12 | 0 | 0.833 | 0.917 | 0.917 | 0.854 | 0.869 | n/a | 0.000 | 243.3 | 0.62 | 1.26 |
| negative | 12 | 0 | 12 | n/a | n/a | n/a | n/a | n/a | 1.000 | 0.000 | 242.7 | 0.62 | 0.91 |

Exact identifiers work well; vocabulary-disjoint paraphrases and selecting the current decision are much weaker. Every negative returned related material rather than an abstention. These are published limitations, not failures hidden by a score gate. Porter stemming, prefix matching, and allowed generic words can still retrieve a paraphrase gold entry despite the authoring vocabulary rule.
