Embeddings

CortexDB calls an embedding service on every event ingest (one vector per chunk) and on every recall query (one for the query, plus N for HyDE-expanded queries). Embeddings are the single largest line item in the inference bill for most workloads, and vector recall does not work without a real provider configured.

The core constraint

The embedding service's output dimension must match the engine's vector index dimension. They are configured separately. A mismatch between the on-disk index width and the configured model is startup-fatal ("vector index dimension mismatch: on-disk index at /data/vector is 1536-dim but the configured embedding model is 3072-dim … refusing to serve broken semantic recall"). Upgrade from earlier builds, which served such a store with empty recall results.

# cortex.toml
[engine]
vector_dimensions = 1536

export CORTEX_EMBEDDING_MODEL=text-embedding-3-small
export CORTEX_EMBEDDING_DIMS=1536

Choosing where embeddings run

The two common self-hosted setups are equally supported and behave identically from the API's point of view; they differ in where the model runs and therefore in what the machine needs.

• OpenAI API, text-embedding-3-small (1536 dims). The image default: set CORTEX_EMBEDDING_API_KEY and nothing else. Nothing runs locally, the same key can serve the answer and enrichment lanes, and it costs fractions of a cent per thousand tokens. Choose this on modest hardware (a laptop, no GPU, limited RAM or disk) or whenever you want the fewest moving parts.
• Local Ollama, nomic-embed-text (768 dims). Free per call and the text to embed never leaves the box, at the price of running an embedding model yourself. Choose this on a high-end machine (spare CPU and RAM, ideally a GPU) or when vectors must stay on-premises. Expect ingest and recall latency to track the host's CPU on machines without a GPU.

Decide before the first write: the provider, model and dimensions are pinned to the data directory (below), and changing them afterwards is a deliberate offline rebuild.

Provider selection

export CORTEX_EMBEDDING_PROVIDER=<empty> | mock | cohere | ollama

The provider pin

On first boot the server writes an embedding_provider.pin file (recording provider:model:dims) into the data directory. On every later start, the configured provider is checked against the pin — a mismatch is startup-fatal (FATAL: embedding provider mismatch: data dir /data is pinned to 'openai-http:text-embedding-3-small:1536' but the configured provider resolves to …), and a refused boot leaves the data directory byte-identical. A missing API key can no longer silently downgrade a real corpus to mock vectors. The pin is not a dead end: switching provider or model on an existing store is the supported procedure in the next section.

Changing the embedding model on an existing store

Vectors from different models are not comparable, so a model change means re-embedding the corpus. That is a supported offline procedure on the same data directory. It takes two deliberate flags, and the destructive one is spent after one boot:

# 1. stop the server, then start it ONCE with the new model plus both flags
docker run -d --name cortexdb -v cortexdb-data:/data \
  -e CORTEX_EMBEDDING_MODEL=text-embedding-3-large -e CORTEX_EMBEDDING_DIMS=3072 \
  -e CORTEX_EMBEDDING_API_KEY=$OPENAI_API_KEY \
  -e CORTEX_EMBEDDING_ALLOW_REPIN=1 -e CORTEX_EMBEDDING_REBUILD_VECTORS=1 \
  cortexdb/cortexdb:v0.12.8
# boot log: "DESTRUCTIVE VECTOR REBUILD … DROPPING chunk (/data/vector, 1536-dim), entity (…)
#            Recreating them EMPTY at 3072-dim"

# 2. refill the chunk vectors from the WAL with the new model (idempotent, resumable)
curl -X POST localhost:3141/v1/admin/index-audit -H "Content-Type: application/json" \
  -d '{"scope":"org:acme","repair":true,"vectors":true}'
# repeat per scope until missing_vectors_total is 0; vector recall returns nothing until then (BM25 only)

# 3. remove the flags from your unit file / compose; confirm with an audit
curl -X POST localhost:3141/v1/admin/index-audit -H "Content-Type: application/json" -d '{"scope":"org:acme"}'

What the server guarantees:

• Never automatic. A dimension mismatch without both flags still refuses to boot. ALLOW_REPIN alone is not enough for a dims change (vector index dimension mismatch … refusing to serve).
• One-shot. The rebuild writes /data/embedding_rebuild.consumed; while it exists the flag "authorizes NOTHING" on later boots (logged as a warning), so a flag left in a unit file is harmless.
• Refused for mock. A lost credential looks like a switch to mock embeddings; that is refused with the data untouched.
• Honest about completeness. /data/vector_rebuild_pending.json marks the store INCOMPLETE and the audit response (vector_rebuild_pending, vector_families) lists each family with its own refill instruction: chunk is refilled by the audit repair; entity, derived and summary vectors are written by enrichment and the layer builds, which must be re-run at the new width (the audit never writes those and does not report them missing). Delete the pending file once the families are back, or to accept the store as it is, and embedding_provider.repin.json after re-embedding.
• ?repair=true without "vectors": true repairs rows only and leaves the vectors empty.

The old alternative, export → fresh data directory on the new provider → import, still works and needs no offline window on the old store. An online re-embed job (dual-embed during the job, atomic swap, no BM25-only window) is not available.

When the API key is missing

An explicitly configured embedding endpoint without a credential is a boot refusal, not a fallback: FATAL: an explicit CORTEX_EMBEDDING_URL requires CORTEX_EMBEDDING_API_KEY, OPENAI_API_KEY, or LLM_API_KEY and the process exits. The image bakes an OpenAI embedding URL, so this includes the plain docker run cortexdb/cortexdb with no key. Mock vectors are meaningless search, and the data directory is pinned to whatever wrote it, so a store filled with mock vectors could never be pointed at a real provider afterwards; refusing is the honest failure. CORTEX_EMBEDDING_PROVIDER=mock is the explicit opt-in (384-d deterministic vectors, development only); a mock store reports degraded: true on GET /v1/admin/ready.

A data directory pinned to mock must be rebuilt on a real provider (see the migration section above).

The full env-var surface

Choosing a model

The local rows assume a machine that can run the model comfortably: spare CPU and RAM, ideally a GPU. On modest hardware the OpenAI default is the better trade (see Choosing where embeddings run); the accuracy gap of the local models is small, the latency gap on a weak host is not.

Batch and retry tuning

The service packs pending requests into provider calls of up to CORTEX_EMBEDDING_MAX_BATCH_ITEMS items. Larger batches improve throughput but raise the latency of the first request in a batch.

The response-size clamp. Each provider response has to fit the OpenAI-compatible client's byte limit, so the client caps each call at floor(12 MiB / (24 × dimensions + 128)) texts: 340 at 1536 dimensions, 170 at 3072, 2006 at 256. A larger CORTEX_EMBEDDING_MAX_BATCH_ITEMS is clamped to that on every call, and the server logs one warning at startup:

WARN cortex_embedding: CORTEX_EMBEDDING_MAX_BATCH_ITEMS=2048 clamped to 340: one response for 2048 x 1536-dim vectors would exceed the embedding client's response byte limit; each provider call carries at most 340 texts configured=2048 effective=340 model="text-embedding-3-small" dims=1536

At 3072 dimensions the image's own 256 is above the limit, so the warning (clamped to 170) appears with no setting of yours; it is harmless. A long document makes more, smaller calls.

Local-first with Ollama

ollama pull nomic-embed-text

export CORTEX_EMBEDDING_URL=http://localhost:11434
export CORTEX_EMBEDDING_MODEL=nomic-embed-text
export CORTEX_EMBEDDING_DIMS=768
export CORTEX_EMBEDDING_API_KEY=ollama          # non-empty required

# cortex.toml — match the dim
[engine]
vector_dimensions = 768

Expect ~30 ms / embedding on a modern CPU. See Self-hosting defaults for the full content-only setup.

Caching

HTTP embedding services are wrapped in an in-process LRU keyed by (model, text_hash). The cache survives the process lifetime; it's lost on restart, and its size is not env-configurable (~10K entries compiled default). Highly effective for re-embedding the same text (eval reruns), near-useless for cold ingest.

Diagnostics

On startup the binary logs which service it picked:

info  Using HTTP embedding service model=text-embedding-3-small dims=1536
info  Using Ollama local embedding service model=nomic-embed-text dims=768 url=http://localhost:11434
info  Using MOCK embeddings (explicitly requested via CORTEX_EMBEDDING_PROVIDER=mock) dims=1536

Grep for "embedding service" after changing config, and check GET /v1/admin/ready — it reports the pinned provider and degraded: true when on mock. Recall failures are usually a missing API key or a dim mismatch.

Next steps

• LLM & Answer Generation — the extraction and answer LLMs
• Recall Tuning — how embeddings feed the recall pipeline
• Storage & Cluster — the HNSW index that consumes these vectors