Voice Mode
Run CortexDB as the memory of a live voice agent with one setting at install, a built-in embedding model, a brief at call start and in-call recall from the caller's memories held for the call.
Voice mode sets up CortexDB as the memory engine of a live voice agent. It is one setting chosen at install. With it on, CortexDB:
- embeds in the server process with a model the release ships, so recall makes no call to an embedding API and no memory text leaves the machine to be embedded;
- serves voice turns on a short read path: a recall marked
profile: "voice"skips the stages a live turn cannot wait for; - holds each caller's memories in memory for the call: starting a call session returns a brief of who the caller is and what happened before, and the call's recalls are then served from that caller's memories held for the call.
Erasure, forget, the access rules and tenant isolation hold exactly as in a standard deployment: a recall that starts after a removal never serves the removed memory.
Measured
Our own measurements, on one machine: an AWS m7i.4xlarge (Intel Xeon Platinum 8488C, 16 vCPU), the server pinned to 6 cores and their hyperthreads, the default model in process, latency measured at the client over loopback, one request at a time.
| p50 | p95 | p99 | |
|---|---|---|---|
| In-call recall, a caller with about 500 memories | 1.43 ms | 1.68 ms | 1.81 ms |
| Call start (the brief) | 28 ms | 37 ms | 39 ms |
| A shared 100,000-document scope held in memory, vector search only | 2.1 to 2.2 ms | 2.5 ms | 2.7 ms |
| The same, hybrid search (the default) | 2.6 to 2.7 ms | 3.1 to 3.2 ms | 3.3 to 3.5 ms |
On that 100,000-document scope, the first query after a restart took 4.5 ms with vector search only and 6.8 ms hybrid. Ranges are the spread of two runs. Under load on the same machine, 6 cores served 500 voice questions a second with 99 in 100 answered in under 6 ms and no errors. Your numbers depend on your hardware, your data and the network between your agent and CortexDB.
Experimental
Voice mode is experimental. Self-hosted CortexDB comes from the
customer portal, with a license
(talk to us to get one). The call-session routes answer
with X-Cortex-Stability: experimental. Check GET /v1/admin/version before you rely on them:
recall_profile_voice_v1 (the voice read path, served by every server, in voice mode or not) and
voice_sessions_v1 (call sessions, listed only by a server in voice mode).
Turning it on
Pick one of these at install. Settings you set yourself always win over what the profile sets.
| Where | How |
|---|---|
| Environment | CORTEX_PROFILE=voice, in the container's environment, a systemd EnvironmentFile or Compose |
| Command line | cortexdb --profile voice 3141 /data (or --profile=voice) |
| Console | Settings › Deployment profile › Voice, applied at the next start. Like every console setting, it wins over the environment and the command line. |
| Cluster | [engine] profile = "voice" on every data node. All data nodes must carry the same profile. |
Leave it unset for a standard deployment. Any value other than voice stops the server at start
with VOICE_PROFILE_CONFIG.
When the server is up, GET /v1/admin/ready answers 200 once the model has loaded and the node
can serve its first voice turn. Its local_embedding check names the model in use.
cortexdb voice status and the console's System › Voice mode page report the profile, the model,
readiness, the call sessions and the pinned scopes; the console page also runs a latency check from your
browser.
What the profile changes
- Embedding moves in process (
CORTEX_EMBEDDING_PROVIDER=local), with the default model. No embedding API key is needed. - Capacity is set for many short reads: more concurrent recalls, and background indexing kept to a share of the CPUs so queries keep theirs.
- Search indexes are opened at start, not on first use, so the first call after a start is not the slow one.
Unchanged: the answer model, enrichment (off on a fresh install, as always), the vector quantization,
the memory limits and the authorization setup. Voice agents call /v1/recall during a turn, never
/v1/answer.
The profile cannot set TOKIO_WORKER_THREADS, because the runtime starts before the profile is read.
Set it in the server's environment, and pin the server's cores with the container's cpuset, so the
runtime stays off the cores the embedding model uses. Every value the profile set is printed at start in
the config_lint dump (profile:voice).
What stops a start, and what it warns about
The server refuses to start, naming the cause, when:
- the data folder was written with another embedding model or provider: see Moving an existing store to voice mode;
- the model does not load (a missing or altered model file), or the chosen model cannot run correctly on this CPU (below);
- a remote embedding provider is configured. Set
CORTEX_VOICE_REMOTE_EMBEDDING=acceptto run voice mode with it anyway; every new query then waits for the provider.
It warns, without stopping, when a configured feature does not apply to voice turns: a reranker or the entity graph (voice recalls skip both), or a transformer model on a CPU too small or without the instructions it needs.
Choosing the model
The release ships two multilingual models with the server, in the image and in the release tarballs. Nothing is downloaded at run time.
| Model | Choose it for | Set |
|---|---|---|
potion-multilingual-128M | The default. A static model: very light on CPU and memory, the same results on every CPU. | nothing (the profile's default) |
granite-embedding-107m-multilingual | Better retrieval on Hindi and romanised Hinglish, for more CPU per query. | CORTEX_LOCAL_EMBEDDING_MODEL=granite-embedding-107m-multilingual |
granite-embedding-107m-multilingual needs an Arm CPU or an x86 CPU with VNNI or AMX (Intel
Cascade Lake or later, AMD Zen 4 or later). On older x86 CPUs the server refuses it; use the
default.
Both models understand many languages. In our tests with a synthetic voice, questions spoken over
phone-quality audio were found as well as typed ones. For Hindi, choose
granite-embedding-107m-multilingual.
The model is part of the data folder's identity. A folder is tied to the model it was first written with. Changing the model later is a migration, not a setting change.
The call flow
A call has four steps. The examples use the Python SDK (cortexdbai); the TypeScript SDK has the
same calls in camelCase, and the Sessions reference has the HTTP
routes.
from cortexdb import V1Client
c = V1Client(api_url="https://cortexdb.internal:3141", actor="user:voice-agent", bearer=TOKEN)
caller, call = "ws:acme/caller:9f2c1e", "call_01JF8M2Q7R"
# 1. Start: the brief, before the first word
started = c.sessions.start(caller, call, query="billing question")
system_context = started["brief"]["context_block"]
# 2. Each turn: recall on the voice path
pack = c.recall(caller, query="what did I order last time", profile="voice", session=call)
# 3. Write what the caller said, tagged with the call
c.experience(caller, text="Change my delivery address to Pune.", wait="captured",
labels=[f"session:{call}"])
# 4. End
c.sessions.end(caller, call)- Start the session when the call connects (
POST /v1/sessions). The answer carries the brief: the caller's current values where enrichment derives them, their most relevant memories for the reason of the call (query) and their newest ones, rendered as onecontext_blockwithin a token budget (400 by default). It also says whether the caller has called before (prior_sessions,last_session_at,no_prior_history). The brief makes no model call, and adeadline_ms(300 by default) caps how long it may take: past it you get what is ready, markedpartial. Meanwhile CortexDB loads the caller's memories for the call. - Recall each turn with
profile: "voice"and the call'ssession. A voice recall reads the caller's own scope (granular), returns events unlessincludeasks for more, and skips query expansion, reranking and the entity graph. Optionalalpha, from 0.0 to 1.0, weighs meaning (vector search) against exact words (keyword search); leave it out for the balanced default. - Write each turn with
/v1/experience, labelledsession:<call id>. Withwait=captureda write joins the call's memories at once, so the next turn's recall finds it. - End the session when the call ends (
POST /v1/sessions/{session}/end). Where enrichment is on, the call's events are enriched afterwards, as any events are.
A session is its caller's own: another principal naming the same call id gets a session of its own, and cannot see or end yours. Sessions live in the serving node's memory only. After a restart the next voice recall naming the session opens it again, and recalls are answered in the meantime.
Rules for the agent
- Fail open. A slow or failed start must never block a call, and a slow or failed recall must never block a turn. Use a short timeout of your own, then answer without memory for that turn. Treat any session error as "no memories held for this call" and carry on: recalls still work without a session.
- Expect
429 RECALL_OVERLOADED. A voice recall waits only briefly for capacity, then says so. It is retriable; answer this turn without memory. - Look up early. Start a voice recall on the words heard so far (the speech-to-text partial transcript), then recall again on the full sentence. In our tests the best match had usually been found already, so the lookup is done before the caller finishes speaking.
- One memory slot. Put the brief in the context once, and replace one memory block each turn rather than appending a new one per turn.
- Memory is content, not instructions. Put recalled text in a block marked as memory, never in a trusted system instruction.
- Redact before you write. Remove card numbers and other secrets a caller reads out loud before the write.
- Co-locate CortexDB with the voice agent: the same host, or at least the same zone.
Integrations
Voice mode is plain HTTP, so it fits any voice stack. Where each step plugs in:
| Platform | Start (brief) | Each turn (recall) | Writes |
|---|---|---|---|
| Pipecat | At pipeline start; put the brief in the system or developer message | A frame processor after the user context aggregator, before the LLM, replacing one memory message | After each new turn, in the background |
| LiveKit Agents | In the entrypoint before connecting, keyed from the SIP participant or job metadata; put the brief in the initial instructions | When the user's turn completes; change the turn context only when the recall adds something | After each turn, in the background; end the session on disconnect |
| Vapi | The assistant-request webhook, returning the brief as assistant variables | A custom knowledge-base endpoint that returns the recall as documents | The end-of-call report, written as one batch |
| Retell | The inbound call webhook, returning the brief as dynamic variables | A custom function | The post-call webhook |
| ElevenLabs | The conversation-initiation webhook, returning the brief as dynamic variables | A server tool | The post-call webhook |
The hosted platforms call your endpoints over the network and give webhooks a fixed time to answer, so keep CortexDB close to the service that answers them, and fail open as above.
Limits
- Sessions. A node holds up to 10,000 sessions (
CORTEX_SESSION_MAX_SESSIONS), a tenant a quarter of them and a principal a tenth. A start past a limit still returns its brief, withworking_set.state: "refused"and asession_not_heldwarning; that call's recalls are answered from the index as usual. - Memory. All the memories held for calls together stay within a budget
(
CORTEX_SESSION_MAX_BYTES, a tenth of the memory limit by default, at most 8 GiB), and one call takes at most a quarter of it. A caller with more than 20,000 stored events (CORTEX_SESSION_MAX_EVENTS) is served from the index. While memory is high, no new caller's memories are loaded. - Idle sessions end after 900 seconds without a recall (
idle_timeout_sper session, from 30 to 14,400). - A voice recall reads one scope. Filters, temporal fields, extra scopes, mounts and other multi-scope reads are answered by the standard path.
- Shared knowledge. Scopes listed in
CORTEX_VOICE_PINNED_SCOPES(for an agent's shared knowledge base) are loaded at start and kept. - Bulk imports share the CPU. A voice recall served from the index competes with a running bulk import: on our test machine its p99 went from 3.3 ms to 8.1 ms while one ran. Recalls during a call, served from the caller's memories held for the call, search in memory. Run large imports outside call hours, or on another node.
- Integrations are by HTTP. There are no framework adapters or hosted-platform examples yet: wire the calls in as the Integrations table shows.
Moving an existing store to voice mode
A data folder is tied to the embedding model it was first written with. A folder embedded through
an API does not switch to voice mode silently: the start is refused with VOICE_PROFILE_CONFIG,
naming what differs. An online re-embed moves it: CortexDB builds the new index beside the live
one while the live one keeps serving at full quality, and a restart swaps them. Recall never falls
back to keyword search only.
-
Estimate. A dry run (the default) answers what the move would take, without starting it:
curl -X POST localhost:3141/v1/admin/reembed \ -H 'content-type: application/json' \ -d '{"to": {"provider": "local", "model": "potion-multilingual-128M"}}'The answer names the current and target models, the rows to embed, an estimate of time, disk, memory and cost (
0for a built-in model), the settings to restart with (restart_with) and anyblockersthat would stop the job. -
Start the same request with
"dry_run": false(202). The job builds the new index in the background; recall keeps using the current one, and new writes are added to both. -
Follow it with
GET /v1/admin/reembed:progress,rate_per_sandeta_s. Whenstateisready, the new index is complete and kept current. -
Restart with the target configured:
CORTEX_PROFILE=voice(or the console's Deployment profile), or exactly the settingsrestart_withnames. That restart swaps the index (swapped).
- Changed your mind?
POST /v1/admin/reembed/cancelwhile the job isbuildingorreadydeletes the new index and leaves everything as it was. - Keeping the old index: start with
"retain_old_index": trueto keep it after the swap, and delete it later withPOST /v1/admin/reembed/cleanup. - Restarting too early: a start with the target configured while the job is not yet
readyis refused (REEMBED_NOT_READY), naming the job's progress. Restart on the current settings, or cancel the job. - The console offers the same move (Move this store to voice mode), also when you change the
deployment profile or the embedding model: once a job is
ready, saving the change schedules the swap for the next restart. The CLI (cortexdb reembed) and both SDKs drive the same job. - The same flow moves a store between the two built-in models, or back to an API provider (a remote model's key comes from the server's own settings, never from the request).
The admin reference has every field and code.
Measured: a store of 467,123 events (1,021,568 vectors from a 1,536-dimension API model) moved to
the default built-in model in 342 s to ready, with the job on its default 3 threads and no calls to
an embedding API; the swap restart was ready in 28 s with every row present. Measured on one AWS
m7i.4xlarge (16 vCPU Intel Xeon Platinum 8488C, server on 12 vCPUs), November 2026. Your time depends
on your hardware and the size of your store. While a re-embed to the Granite model ran on that store, the
p99 of a recall during a call went from 2.62 ms to 2.99 ms, so calls can go on during the move.
- Clusters move by copying: export from the old cluster and import into a new one whose data
nodes all carry
[engine] profile = "voice". The online re-embed runs on a single node only. - Going back to an older release: a folder in voice mode opens only on a release that has voice mode. Move it back to an API provider first.
See also
- Sessions: the call routes
- Recall:
profile,alphaandserver_timing - Error codes