v1.0.11 formal benchmark — consolidated#
Single-page reference of all hard-locked retrieval + graph-quality numbers for the v1.0.11 release. Sources are linked per-row. Numbers are reproducible via the listed CLI commands; no LLM-rephrasing or estimation.
Headline numbers#
| metric | value | n | source |
|---|---|---|---|
| LongMemEval-S R@5 (v0.3-A K=5 sweet-spot) | 76.77% | 99 | baseline.json:v03_fused_a_rrf_optimal |
| LongMemEval-S R@5 (v0.3-B BGE-reranker) | 73.74% | 99 | baseline.json:v03_fused_b_reranker |
| LongMemEval-S R@5 (v0.2 hybrid) | 67.68% | 99 | baseline.json:hybrid |
| LongMemEval-S R@5 (cosine alone) | 31.31% | 99 | baseline.json:cosine |
| Production-stack R@5 (vault-search-fusion, tuning set) | 85.39% | 89 | Production retrieval-stack v2 RRF ADR |
| Production-stack R@5 (vault-search-fusion, held-out) | 69.66% | 30 | same |
| Production-stack R@5 (vault-search-fusion, avg) | 77.50% | 119 | same |
| Memgraph typed-coverage | 53.6% | 13,307 | vault-doctor live read |
| Layer-3 retrieval latency (vault-ko-query, default route) | 1.20s mean | 3 | vault-continuous-eval retrieval-latency axis |
| --semantic vs --semantic-rrf speedup | 5.65× | 5 queries × 2 routes | this session A/B benchmark |
| B-7 LLM-extract FP rate | <5% | 45 random samples | this session manual review |
| Critic Cohen's κ (B-8 production-flip) | 0.708 | 100 bullets | B-8 production-flip ADR |
Reproduce#
# LongMemEval-S — full 99-Q sweep, all v0.3 variants
vault-eval-regression --v03 # ~15-20 min
# LongMemEval-S — fast 20-Q subset
vault-eval-regression --v03-fast # ~3-5 min
# Continuous-eval rollup (8 axes, weekly)
vault-continuous-eval --json | jq
# vault-doctor 8-axis health snapshot
vault-doctor --json
# vault-ko-query A/B benchmark
/tmp/semrrf-benchmark.py # 5-query A/B (semantic vs semantic-rrf)
How the numbers compare to public references#
| system | R@5 | notes |
|---|---|---|
| MyForge Vault v1.0.11 (LongMemEval-S sweet-spot) | 76.77% | 99-Q, K=5, RRF fusion, no reranker |
| MyForge Vault v1.0.11 (production-stack, avg) | 77.50% | 119-Q (89 tuning + 30 held-out), agentmemory+vault-search RRF |
| agentmemory standalone (cherry-picked v0.9.21) | 76.40% | reported on same setup, 89-Q |
| vault-search standalone (Memgraph + bge-m3) | 55.06% | reported on same setup, 89-Q |
| BEIR-suite cosine-only baselines | ~30-50% | varies per dataset; vault corpus is in the same ballpark on cosine-only (31.31%) |
Two interpretations of "best number": - LongMemEval-S academic benchmark: 76.77% R@5 (K=5 sweet-spot, no reranker) - Own production stack on own queries: 77.50% avg R@5 (RRF of vault-search + agentmemory)
These are independent measurements on different benchmarks — both legitimate.
What changed since v1.0.10#
- Typed-coverage: 2.0% → 53.6% (+26.6×) via 132-batch / 9-wave / 16-parallel subagent fanout, $0 cost. Quality audit on 45-sample random review: ~95% accurate, FP <5%.
vault-ko-querydefault route: now--semantic-rrf(was--semantic). 5.65× faster latency, same fallback chain.vault-doctor: new 8-axis unified health-check, 0.1s, color-coded, exit-code protocol.vault-continuous-eval: 6 → 8 axes (typed-coverage + retrieval-latency added).- Cron mutex-coverage: 88% → 100% (4 entries wrapped in
flock -n).
Open follow-ups (not in v1.0.11)#
- Typed-graph second-pass with context: feed wiki-sentence + 1-hop neighbors → expected 53.6% → ~75%. Same fanout pattern, ~3h.
- Cross-encoder reranker as Layer-4 on top of RRF top-20 → expected R@5 77.5% → ~85%+. Self-hosted bge-reranker-v2-m3, ~3-4h.
- Broken-wikilink cleanup: 35 broken targets currently. Mostly renamed/archived files; needs triage.
Audit-log discipline#
Every reported number above MUST be:
1. Stored in a structured file (baseline.json, vault-eval-regression.json, continuous-eval-*.md, audit-log JSONL).
2. Reproducible via a documented CLI command.
3. Versioned (locked-at timestamp).
If you can't run the command and get the same number back, the number doesn't go in this report.