CortexDB v1 Architecture: The Experience Layer for AI Agents

Abstract

CortexDB v1 treats every interaction, document, event, or observation as an immutable experience appended to a write-ahead log, then asynchronously derives five layered views — Events, Episodes, Facts, Beliefs, Understanding — that recall and answer endpoints query against. Every record is bi-temporal (carries both when it was true and when the system learned it). Every request is scoped to a hierarchical path and gated by a four-tier capability stack with PASETO v4 public-token identity.

1. Design constraints

1. No information loss on the write path — captured bytes are retained verbatim; summarization, dedup, and structuring happen downstream and can be re-run.
2. Bi-temporal correctness — answer both "what is true now" and "what did we know yesterday."
3. Scope as a first-class concept — multi-tenant, multi-user, and multi-agent all reduce to one primitive: a hierarchical path with ancestor/descendant semantics.
4. Capability-based authorization — every denial cites the deciding tier and the missing capability.
5. Async-by-default, synchronously-completable — writes return 202 in under 10 ms; clients can opt into wait=indexed.

2. The surface

The daily-driver API is five endpoints:

POST /v1/auth/signup     anonymous PASETO (dev_local, no API key set)
POST /v1/experience      append an experience (the only write)
POST /v1/recall          return a StratifiedPack (retrieval)
POST /v1/answer          recall + LLM answer + citations
POST /v1/forget          delete with an audit note (selective or GDPR)

Layer-direct reads (/v1/events, /v1/facts, …) exist for inspection; /v1/recall is the standard read path because it merges across layers. Identity is PASETO v4 public tokens; every request carries Authorization: Bearer + X-Cortex-Actor (must equal the token's sub, else 401 actor_mismatch).

3. The five layers

• Events — the WAL; every POST /v1/experience becomes one immutable event ({ id, scope, modality, content, context, observed_actor }), idempotency-keyed so a resubmit replays the original response.
• Episodes — the segmenter's chronologically-contiguous spans (a meeting, a thread, an incident).
• Facts — subject/predicate/object triples, each with valid_from/valid_to (true in the world) and recorded_from/recorded_to (when the system knew). The largest single contributor to the benchmark scores (−22 pp if disabled).
• Beliefs — facts aggregated into confidence-weighted claims, explainable via GET /v1/beliefs/why.
• Understanding — the synthesized concept layer (named concepts, themes, relationships); the slowest and most expensive, gated by understanding.synthesize.

4. The five-stage lifecycle

CAPTURE     → POST /v1/experience returns 202 + id once the WAL append commits.
EXTRACT     → LLM extractor pulls triples into Facts on the next scheduler tick.
RECONCILE   → Conflicts resolve under bi-temporal supersession (newest valid_from wins;
               the older record's valid_to is set to the new one's valid_from).
FORGET      → Selective forget across derived layers (derived_only / redact_events / gdpr). Audited.
CONSOLIDATE → Synthesizer builds Understanding concepts per topic per scope.

Stage 1 is synchronous (the 202 means durable in the WAL); stages 2–5 are async — pass ?wait=indexed (or consolidated) to block on a specific stage.

5. Bi-temporal storage

Every record carries valid_from/valid_to (true in the world) and recorded_from/recorded_to (when the system learned it), supporting four query modes from the same data as direct typed-store lookups (no LLM, no scan): Now (valid_to and recorded_to null), As-of (valid_from <= t < valid_to), As-known (recorded_from <= t < recorded_to), and History (the full supersession chain). Most memory systems omit the recorded_* axis and cannot answer "what did we believe at the time."

6. Scopes

A scope is a hierarchical path (org:acme/dept:eng/team:platform/user:alice) with three semantics: addressing (every experience is written to one scope), read (holistic traverses up, descend down, granular/local the named scope only), and policy (capabilities granted at any node inherit down the path). There is no flat tenant_id field in the v1 envelope.

7. The four-tier capability stack

Evaluated outer-to-inner: Deployment → Tenant → Scope → Actor. A more-specific inner allow can override an outer allow; an outer deny is final. Anonymous signup tokens carry the standard read/write set but not diagnostics.read, auth.mint, or forget.gdpr. Denials are explicit (error_code POLICY_DENIED with details.decided_by_tier + a reason), and GET /v1/policy/effective?actor=&scope= returns the allow/deny sets for any actor-scope pair.

8. The auth model

9. Storage and durability

The WAL is the system of record; every derived layer can be rebuilt from it — a corrupted index is an operational annoyance, not data loss.

10. Observability

Every authenticated response carries X-RateLimit-Limit / -Reset, X-Cortex-Token-Expires-In / -At, X-Cortex-Request-Id, and a Warning header when token expiry ≤ 72h. GET /v1/lifecycle/stream (SSE) emits a payload per stage transition, with Last-Event-ID catchup.

11. Trade-offs

• Storage cost — the raw event stream is ~1.3× the storage of a rewrite-based memory layer; the price of bi-temporal correctness.
• Read latency — holistic recall p50 ~500 ms (the hybrid BM25 + HNSW + graph + reranker stage); granular skips half the pipeline (~80 ms) for a single-scope lookup.
• Async derivation — Facts in seconds, Beliefs/Understanding minutes-to-hours; a recall right after a write returns the raw event but may not yet reflect derived knowledge (?wait=indexed).
• LLM dependency on the read path — /v1/answer and synthesis need an LLM, but /v1/recall degrades gracefully and still returns the stratified pack.

12. What ships in v1 vs what comes next

13. Conclusion

CortexDB v1 is a shipped, benchmarked production memory layer for AI agents — five derived layers from one source of truth, bi-temporal across both axes, scoped hierarchically, gated by an explicit capability stack, callable in one anonymous-signup curl. See Core Concepts for the reference treatment of each component.