# How we measured

Every figure on harnesslink.sh is listed here, with how it was measured and what it does not show. Unless a section says otherwise, these are our own measurements, on one machine, from our own sessions. They record what happened in those runs; your numbers will differ.

Each section names its source: an entry in the architecture decision log (`docs/architecture/decision-log.md` in the HarnessLink repository), where the full record is kept.

## Folding a stack trace

**The figure.** A real Python RecursionError went from 4,215 to 136 tokens.

**How it was measured.** The gateway caps a tool result that is too large before it reaches the model. Where the cap would cut a stack trace, the code-aware cap groups the trace into frames of one to three lines. It then folds repeated frames and cycles, library frames and later frames from your own code, and says how many it folded. We ran it on one real RecursionError and compared the size before and after.

**Limits.** One trace. A recursion error repeats the same frames many times, so it folds unusually well. The fold runs only where the size cap would already cut the result; anything smaller passes through unchanged. Every line kept is an input line, unchanged, and the full original can be fetched with `context_expand`.

**Source.** ADR-93, "Code-aware size capping".

## Folding a test log

**The figure.** A mixed Rust, Node and Python test log went from 5,704 to 1,564 tokens.

**How it was measured.** The same code-aware cap as above, run on one real test log with output from Rust, Node and Python tests, compared before and after.

**Limits.** One log. A log with fewer repeated traces folds less. As above, the fold runs only where the size cap would already cut the result.

**Source.** ADR-93, "Code-aware size capping".

## Finding code without reading whole files

**The figure.** 75% fewer tokens than grep plus whole-file reads, on 10 real questions.

**How it was measured.** Ten real questions about code were answered two ways: with the `code_map` tool, and with grep plus reading the whole files it found. `code_map` keeps a local index of each project's files and symbols and answers four kinds of question: an overview, one file's outline, where a symbol is defined, and where it is used. We compared how many tokens each way returned. In the same tests, `code_map` missed no references for 19 names, and built the first index of a 6,700-file repository in under 2 seconds.

**Limits.** Ten questions, chosen by us, in one repository. We compared the size of what each way returned, not how well an agent then did its task. The index reads code with simple per-language rules, not a full parser.

**Source.** ADR-91, "Code map: a local symbol index served as one MCP tool".

## Leaving out unused tool descriptions

**The figure.** The hub forwarded 447 tools into every session. Codex sends them in full, about 155k tokens per call. Only 8 of 953 sessions used any hub tool.

**How it was measured.** The hub is HarnessLink's MCP server: it forwards the tools of the other MCP servers you connect. On our own machine we counted the tools it forwarded, the size of their descriptions in a Codex request, and how many of our sessions called a hub tool at all. Only 24 of the 429 hub tools were ever called. HarnessLink now hides a forwarded tool in a repository once that repository has enough sessions and the tool has gone unused in them. A hidden tool can still be called by name, and `hnl tools --restore` brings them all back.

**Limits.** One machine, with many MCP servers connected. A setup with fewer servers forwards fewer tools and saves less. The 155k figure is Codex's, because Codex sends every description in full; in Claude Code and omp the gateway's compression already cut the list.

**Source.** ADR-94, "Hub tools pruned per repository from usage evidence; built-in removal only suggested".

## Answering a repeated request from the cache

**The figure.** The second and third identical runs were answered from the cache: 27,698 input tokens saved.

**How it was measured.** The gateway keeps an exact-match cache of model responses. Some clients change parts of every request without changing what it asks, such as fresh item ids or a per-request hash; those parts are masked for the cache key only. In a live run on a Mac, with a build from the repository, omp 18.4 signed in with an Anthropic login sent the same request three times through the gateway. The second and third runs were answered from the cache.

**Limits.** One live run, with one harness. Only an exact repeat is answered from the cache: we chose not to answer similar requests from it, because the answer would belong to a different request. Codex streams over a WebSocket, so its savings come from the provider's own prompt cache, not this one.

**Source.** ADR-34, "The gateway: SuperPi sits in front of every model call" (SuperPi was HarnessLink's earlier name).

## Keeping an idle conversation warm

**The figure.** About 40% of cache-write spend went on writing a prefix again after a pause. In a ten-day replay, the refreshes cost about a fifth of what they saved.

**How it was measured.** Anthropic's prompt cache keeps a conversation's prefix for 5 minutes. A conversation that resumes after a longer pause writes the whole prefix again, at a higher price than reading it. We looked at ten days of our own omp sessions, billed by API key. About 40% of what we spent on cache writes went on writing a prefix again after a pause of 5 to 60 minutes. Of the pauses nobody kept warm, 96% were in omp subagents, whose caches omp does not refresh. We then replayed the same ten days with a refresh every 4.5 minutes, for up to 35 idle minutes. The refreshes cost about a fifth of what they saved. The gateway now does this by default for any idle Anthropic conversation.

**Limits.** One person's sessions, in one harness (omp), on API-key billing, over ten days. The saving comes from a replay of those sessions, not a live measurement. It depends on how often you come back to a paused conversation: in our sessions, the chance of resuming in each five-minute slot was between 0.18 and 0.36 up to 35 minutes, and 0.05 to 0.10 after that. If you rarely come back, the refreshes can cost more than they save. The dashboard shows the net, and setting `gateway.cache.keepWarmMinutes` to `0` turns keep-warm off.

**Source.** ADR-49, "Gateway keep-warm for Anthropic prompt caches, on by default for 35 minutes".

## Finding the right memory

**The figure.** With keyword search and search by meaning together, recall in the first ten results is 0.926; by keyword alone it is 0.806. For questions worded differently from the entry that answers them, it is 0.639 against 0.111. The home page rounds these to 93%, 81%, 64% and 11%.

**How it was measured.** On our memory benchmark: 432 made-up memory entries from six fictional software projects, and 216 questions with the entries that answer them marked in advance. The keyword run uses the product's own local memory search. The full run uses memory search as `memory_search` calls it: keyword search fused with search by meaning (Workers AI embeddings in Cloudflare Vectorize), against a local copy of the cloud service using the live embedding model. Recall in the first ten is the share of a question's answering entries found in the first ten results, averaged over the 180 questions that have an answer. The 0.926 was measured after related entries were grouped by topic; the other full-run figures here were measured just before that change. On the held-out test questions alone, recall in the first ten was 0.917 against 0.783, and 0.583 against 0.083 for reworded questions.

**Limits.** The entries and questions are synthetic, written by AI agents for the benchmark. They are not real developers' sessions, and not our own memory. The same authors marked which entries answer each question. A question with several answering entries gets partial credit, so 0.926 is an average, not a count of questions answered. Every question with no answer still returned results: nothing says "no match". Search by meaning needs the cloud; the local index, the context hook and team search use keywords only. In this run, search by meaning took about half a second at the median, through a development proxy, against under a millisecond for the local index.

**Source.** ADR-102, "Memory search by meaning: Workers AI embeddings in Vectorize, fused with keyword search", and ADR-106, "Consolidate by topic, forget one entry everywhere (M3)". The benchmark, its data and how to run it again: [Coding-memory retrieval benchmark](https://harnesslink.sh/docs/memory-benchmark).
