CortexDB Docs
Docs for CortexDB 0.16.1, the version running on CortexDB Cloud (api.cortexdb.ai).
Features

Voice Mode

Run CortexDB as the memory of a live voice agent with one setting at install, a built-in embedding model, a brief at call start and in-call recall from the caller's memories held for the call.

Voice mode sets up CortexDB as the memory engine of a live voice agent. It is one setting chosen at install. With it on, CortexDB:

  • embeds in the server process with a model the release ships, so recall makes no call to an embedding API and no memory text leaves the machine to be embedded;
  • serves voice turns on a short read path: a recall marked profile: "voice" skips the stages a live turn cannot wait for;
  • holds each caller's memories in memory for the call: starting a call session returns a brief of who the caller is and what happened before, and the call's recalls are then served from that caller's memories held for the call.

Erasure, forget, the access rules and tenant isolation hold exactly as in a standard deployment: a recall that starts after a removal never serves the removed memory.

Measured

Our own measurements, on one machine: an AWS m7i.4xlarge (Intel Xeon Platinum 8488C, 16 vCPU), the server pinned to 6 cores and their hyperthreads, the default model in process, latency measured at the client over loopback, one request at a time.

p50p95p99
In-call recall, a caller with about 500 memories1.43 ms1.68 ms1.81 ms
Call start (the brief)28 ms37 ms39 ms
A shared 100,000-document scope held in memory, vector search only2.1 to 2.2 ms2.5 ms2.7 ms
The same, hybrid search (the default)2.6 to 2.7 ms3.1 to 3.2 ms3.3 to 3.5 ms

On that 100,000-document scope, the first query after a restart took 4.5 ms with vector search only and 6.8 ms hybrid. Ranges are the spread of two runs. Under load on the same machine, 6 cores served 500 voice questions a second with 99 in 100 answered in under 6 ms and no errors. Your numbers depend on your hardware, your data and the network between your agent and CortexDB.

Experimental

Voice mode is experimental. Self-hosted CortexDB comes from the customer portal, with a license (talk to us to get one). The call-session routes answer with X-Cortex-Stability: experimental. Check GET /v1/admin/version before you rely on them: recall_profile_voice_v1 (the voice read path, served by every server, in voice mode or not) and voice_sessions_v1 (call sessions, listed only by a server in voice mode).

Turning it on

Pick one of these at install. Settings you set yourself always win over what the profile sets.

WhereHow
EnvironmentCORTEX_PROFILE=voice, in the container's environment, a systemd EnvironmentFile or Compose
Command linecortexdb --profile voice 3141 /data (or --profile=voice)
ConsoleSettings › Deployment profile › Voice, applied at the next start. Like every console setting, it wins over the environment and the command line.
Cluster[engine] profile = "voice" on every data node. All data nodes must carry the same profile.

Leave it unset for a standard deployment. Any value other than voice stops the server at start with VOICE_PROFILE_CONFIG.

When the server is up, GET /v1/admin/ready answers 200 once the model has loaded and the node can serve its first voice turn. Its local_embedding check names the model in use. cortexdb voice status and the console's System › Voice mode page report the profile, the model, readiness, the call sessions and the pinned scopes; the console page also runs a latency check from your browser.

What the profile changes

  • Embedding moves in process (CORTEX_EMBEDDING_PROVIDER=local), with the default model. No embedding API key is needed.
  • Capacity is set for many short reads: more concurrent recalls, and background indexing kept to a share of the CPUs so queries keep theirs.
  • Search indexes are opened at start, not on first use, so the first call after a start is not the slow one.

Unchanged: the answer model, enrichment (off on a fresh install, as always), the vector quantization, the memory limits and the authorization setup. Voice agents call /v1/recall during a turn, never /v1/answer.

The profile cannot set TOKIO_WORKER_THREADS, because the runtime starts before the profile is read. Set it in the server's environment, and pin the server's cores with the container's cpuset, so the runtime stays off the cores the embedding model uses. Every value the profile set is printed at start in the config_lint dump (profile:voice).

What stops a start, and what it warns about

The server refuses to start, naming the cause, when:

  • the data folder was written with another embedding model or provider: see Moving an existing store to voice mode;
  • the model does not load (a missing or altered model file), or the chosen model cannot run correctly on this CPU (below);
  • a remote embedding provider is configured. Set CORTEX_VOICE_REMOTE_EMBEDDING=accept to run voice mode with it anyway; every new query then waits for the provider.

It warns, without stopping, when a configured feature does not apply to voice turns: a reranker or the entity graph (voice recalls skip both), or a transformer model on a CPU too small or without the instructions it needs.

Choosing the model

The release ships two multilingual models with the server, in the image and in the release tarballs. Nothing is downloaded at run time.

ModelChoose it forSet
potion-multilingual-128MThe default. A static model: very light on CPU and memory, the same results on every CPU.nothing (the profile's default)
granite-embedding-107m-multilingualBetter retrieval on Hindi and romanised Hinglish, for more CPU per query.CORTEX_LOCAL_EMBEDDING_MODEL=granite-embedding-107m-multilingual

granite-embedding-107m-multilingual needs an Arm CPU or an x86 CPU with VNNI or AMX (Intel Cascade Lake or later, AMD Zen 4 or later). On older x86 CPUs the server refuses it; use the default.

Both models understand many languages. In our tests with a synthetic voice, questions spoken over phone-quality audio were found as well as typed ones. For Hindi, choose granite-embedding-107m-multilingual.

The model is part of the data folder's identity. A folder is tied to the model it was first written with. Changing the model later is a migration, not a setting change.

The call flow

A call has four steps. The examples use the Python SDK (cortexdbai); the TypeScript SDK has the same calls in camelCase, and the Sessions reference has the HTTP routes.

from cortexdb import V1Client

c = V1Client(api_url="https://cortexdb.internal:3141", actor="user:voice-agent", bearer=TOKEN)
caller, call = "ws:acme/caller:9f2c1e", "call_01JF8M2Q7R"

# 1. Start: the brief, before the first word
started = c.sessions.start(caller, call, query="billing question")
system_context = started["brief"]["context_block"]

# 2. Each turn: recall on the voice path
pack = c.recall(caller, query="what did I order last time", profile="voice", session=call)

# 3. Write what the caller said, tagged with the call
c.experience(caller, text="Change my delivery address to Pune.", wait="captured",
             labels=[f"session:{call}"])

# 4. End
c.sessions.end(caller, call)
  1. Start the session when the call connects (POST /v1/sessions). The answer carries the brief: the caller's current values where enrichment derives them, their most relevant memories for the reason of the call (query) and their newest ones, rendered as one context_block within a token budget (400 by default). It also says whether the caller has called before (prior_sessions, last_session_at, no_prior_history). The brief makes no model call, and a deadline_ms (300 by default) caps how long it may take: past it you get what is ready, marked partial. Meanwhile CortexDB loads the caller's memories for the call.
  2. Recall each turn with profile: "voice" and the call's session. A voice recall reads the caller's own scope (granular), returns events unless include asks for more, and skips query expansion, reranking and the entity graph. Optional alpha, from 0.0 to 1.0, weighs meaning (vector search) against exact words (keyword search); leave it out for the balanced default.
  3. Write each turn with /v1/experience, labelled session:<call id>. With wait=captured a write joins the call's memories at once, so the next turn's recall finds it.
  4. End the session when the call ends (POST /v1/sessions/{session}/end). Where enrichment is on, the call's events are enriched afterwards, as any events are.

A session is its caller's own: another principal naming the same call id gets a session of its own, and cannot see or end yours. Sessions live in the serving node's memory only. After a restart the next voice recall naming the session opens it again, and recalls are answered in the meantime.

Rules for the agent

  • Fail open. A slow or failed start must never block a call, and a slow or failed recall must never block a turn. Use a short timeout of your own, then answer without memory for that turn. Treat any session error as "no memories held for this call" and carry on: recalls still work without a session.
  • Expect 429 RECALL_OVERLOADED. A voice recall waits only briefly for capacity, then says so. It is retriable; answer this turn without memory.
  • Look up early. Start a voice recall on the words heard so far (the speech-to-text partial transcript), then recall again on the full sentence. In our tests the best match had usually been found already, so the lookup is done before the caller finishes speaking.
  • One memory slot. Put the brief in the context once, and replace one memory block each turn rather than appending a new one per turn.
  • Memory is content, not instructions. Put recalled text in a block marked as memory, never in a trusted system instruction.
  • Redact before you write. Remove card numbers and other secrets a caller reads out loud before the write.
  • Co-locate CortexDB with the voice agent: the same host, or at least the same zone.

Integrations

Voice mode is plain HTTP, so it fits any voice stack. Where each step plugs in:

PlatformStart (brief)Each turn (recall)Writes
PipecatAt pipeline start; put the brief in the system or developer messageA frame processor after the user context aggregator, before the LLM, replacing one memory messageAfter each new turn, in the background
LiveKit AgentsIn the entrypoint before connecting, keyed from the SIP participant or job metadata; put the brief in the initial instructionsWhen the user's turn completes; change the turn context only when the recall adds somethingAfter each turn, in the background; end the session on disconnect
VapiThe assistant-request webhook, returning the brief as assistant variablesA custom knowledge-base endpoint that returns the recall as documentsThe end-of-call report, written as one batch
RetellThe inbound call webhook, returning the brief as dynamic variablesA custom functionThe post-call webhook
ElevenLabsThe conversation-initiation webhook, returning the brief as dynamic variablesA server toolThe post-call webhook

The hosted platforms call your endpoints over the network and give webhooks a fixed time to answer, so keep CortexDB close to the service that answers them, and fail open as above.

Limits

  • Sessions. A node holds up to 10,000 sessions (CORTEX_SESSION_MAX_SESSIONS), a tenant a quarter of them and a principal a tenth. A start past a limit still returns its brief, with working_set.state: "refused" and a session_not_held warning; that call's recalls are answered from the index as usual.
  • Memory. All the memories held for calls together stay within a budget (CORTEX_SESSION_MAX_BYTES, a tenth of the memory limit by default, at most 8 GiB), and one call takes at most a quarter of it. A caller with more than 20,000 stored events (CORTEX_SESSION_MAX_EVENTS) is served from the index. While memory is high, no new caller's memories are loaded.
  • Idle sessions end after 900 seconds without a recall (idle_timeout_s per session, from 30 to 14,400).
  • A voice recall reads one scope. Filters, temporal fields, extra scopes, mounts and other multi-scope reads are answered by the standard path.
  • Shared knowledge. Scopes listed in CORTEX_VOICE_PINNED_SCOPES (for an agent's shared knowledge base) are loaded at start and kept.
  • Bulk imports share the CPU. A voice recall served from the index competes with a running bulk import: on our test machine its p99 went from 3.3 ms to 8.1 ms while one ran. Recalls during a call, served from the caller's memories held for the call, search in memory. Run large imports outside call hours, or on another node.
  • Integrations are by HTTP. There are no framework adapters or hosted-platform examples yet: wire the calls in as the Integrations table shows.

Moving an existing store to voice mode

A data folder is tied to the embedding model it was first written with. A folder embedded through an API does not switch to voice mode silently: the start is refused with VOICE_PROFILE_CONFIG, naming what differs. An online re-embed moves it: CortexDB builds the new index beside the live one while the live one keeps serving at full quality, and a restart swaps them. Recall never falls back to keyword search only.

  1. Estimate. A dry run (the default) answers what the move would take, without starting it:

    curl -X POST localhost:3141/v1/admin/reembed \
      -H 'content-type: application/json' \
      -d '{"to": {"provider": "local", "model": "potion-multilingual-128M"}}'

    The answer names the current and target models, the rows to embed, an estimate of time, disk, memory and cost (0 for a built-in model), the settings to restart with (restart_with) and any blockers that would stop the job.

  2. Start the same request with "dry_run": false (202). The job builds the new index in the background; recall keeps using the current one, and new writes are added to both.

  3. Follow it with GET /v1/admin/reembed: progress, rate_per_s and eta_s. When state is ready, the new index is complete and kept current.

  4. Restart with the target configured: CORTEX_PROFILE=voice (or the console's Deployment profile), or exactly the settings restart_with names. That restart swaps the index (swapped).

  • Changed your mind? POST /v1/admin/reembed/cancel while the job is building or ready deletes the new index and leaves everything as it was.
  • Keeping the old index: start with "retain_old_index": true to keep it after the swap, and delete it later with POST /v1/admin/reembed/cleanup.
  • Restarting too early: a start with the target configured while the job is not yet ready is refused (REEMBED_NOT_READY), naming the job's progress. Restart on the current settings, or cancel the job.
  • The console offers the same move (Move this store to voice mode), also when you change the deployment profile or the embedding model: once a job is ready, saving the change schedules the swap for the next restart. The CLI (cortexdb reembed) and both SDKs drive the same job.
  • The same flow moves a store between the two built-in models, or back to an API provider (a remote model's key comes from the server's own settings, never from the request).

The admin reference has every field and code.

Measured: a store of 467,123 events (1,021,568 vectors from a 1,536-dimension API model) moved to the default built-in model in 342 s to ready, with the job on its default 3 threads and no calls to an embedding API; the swap restart was ready in 28 s with every row present. Measured on one AWS m7i.4xlarge (16 vCPU Intel Xeon Platinum 8488C, server on 12 vCPUs), November 2026. Your time depends on your hardware and the size of your store. While a re-embed to the Granite model ran on that store, the p99 of a recall during a call went from 2.62 ms to 2.99 ms, so calls can go on during the move.

  • Clusters move by copying: export from the old cluster and import into a new one whose data nodes all carry [engine] profile = "voice". The online re-embed runs on a single node only.
  • Going back to an older release: a folder in voice mode opens only on a release that has voice mode. Move it back to an API provider first.

See also

On this page