CortexDB v1 on the Public Memory Benchmarks

Abstract

We report CortexDB v1's scores on the two public, standardized long-term-memory benchmarks for conversational AI: LongMemEval-S (ICLR 2025; 500 questions across six skill categories) and LoCoMo (NAACL 2024; 1,540 QA pairs). On LongMemEval-S, CortexDB v1 reaches 93.8% (469/500) with the server-parity pipeline; on LoCoMo categories 1–4, 86.9% (1,339/1,540) with the same production write-path code. Both runs are single-attempt, with no retry targeting and no gold-oracle leakage; cost and wall-clock are reported per run and reproducible from one command.

1. Why benchmark at all

A long-term memory layer is easy to demo and hard to verify. The interesting questions — does it work across hundreds of sessions, answer temporal questions correctly, cite the right turn out of thousands — only show up at scale on test sets the author didn't design. Two public benchmarks serve this role:

• LongMemEval-S (Wu et al., 2025) — 500 hand-labeled questions across six categories; the "-S" panel averages ~115k input tokens per question.
• LoCoMo (Maharana et al., 2024) — 1,540 QA pairs across very-long conversations (avg. ~9,000 turns).

We use the official datasets, the official evaluator scripts (LLM-judge), and the same write path that ships in production.

2. Headline numbers

3. Methodology

Server parity means every memory is written via POST /v1/experience, every retrieval is via POST /v1/recall / POST /v1/answer, with no question-specific tuning and no oracle access at retrieval time — the same endpoints a paying customer hits.

The pipeline used the public-SaaS defaults: capture raw turns to the WAL; extract subject/predicate/object triples into the Facts layer; reconcile contradictions bi-temporally; recall holistic with include=["events","episodes","facts","beliefs"]; answer with citations.

4. LongMemEval-S results

Cost per LongMemEval-S run: $49.69 total (write-path extraction $18.42, read-path answer + judge $24.71, Cohere rerank $4.56, embedding $2.00) over 2h 2m wall clock, reproducible from benchmarks/longmemeval/RESULTS.md.

5. LoCoMo results

Category 5 ("adversarial", answer = "I don't know") is omitted because its score is refusal-prompt-dependent rather than memory-dependent, and not comparable across systems.

6. Architectural drivers

• Bi-temporal Facts layer (drives temporal-reasoning): every triple stores valid_from/valid_to and recorded_from/recorded_to, so an as-of question hits a typed-store lookup for the correct historical value — where freeform "LLM-rewriting" memory layers collapse this to a single current state at write time.
• Cross-encoder reranker (drives single-session): sorts the top-50 candidates by question-relevance.
• Hybrid recall (drives multi-hop): graph traversal follows entity edges to connect facts that don't share embedding-space neighborhood; disabling the graph stage drops multi-hop by ~13 pp.

7. Ablations

The single largest contributor is the async fact-extraction pipeline (−22.4 pp if removed).

8. Operational characteristics

The write path is a disk append with no LLM call — that's what gives 4 ms p50 and 0% error rate; extraction runs async, decoupled from ingest.

9. Threats to validity

• Benchmark coverage — both benchmarks bias toward conversational settings; 93.8% may not generalize outside conversational memory.
• Model availability — Claude Opus 4.6 and Cohere rerank-v3.5 are commercial; a self-hosted Llama-3.1-70B + BGE-large-en-v1.5 swap reaches 88.4% on LongMemEval-S.
• Single-run variance — LongMemEval-S was replicated six times (April–May 2026), σ = 0.31 pp; the reported number is the median.
• Judge subjectivity — swapping the GPT-4o judge for Claude Opus 4.6 shifts scores ±0.6 pp without changing the ranking.

10. References

Wu et al. (2025), LongMemEval, ICLR 2025 · Maharana et al. (2024), LoCoMo, NAACL 2024 · Chhikara et al. (2025), Mem0 · Packer et al. (2023), MemGPT · Lewis et al. (2020), RAG, NeurIPS 2020.

See also Benchmarking for the reproduction steps.