Embeddings

CortexDB calls an embedding service on every event ingest (one vector per chunk) and on every recall query (one vector for the query, plus N for HyDE-expanded queries). Embeddings are the single largest line item in your inference bill for most workloads.

The core constraint

The embedding service's output dimension must match the engine's vector index dimension. They are configured separately and CortexDB does not check them against each other at startup — a mismatch produces silent recall failures (everything returns 0 results).

The defaults don't match each other. This is a known pitfall: the env-var default targets text-embedding-3-small (1536), and the TOML default targets text-embedding-3-large (3072). Pick one model and set both consistently. The vector_dimensions field accepts only {256, 384, 512, 768, 1024, 1536, 3072} — anything else fails schema validation at startup.

A correct minimal config for text-embedding-3-small:

# cortex.toml
[engine]
vector_dimensions = 1536

export CORTEX_EMBEDDING_MODEL=text-embedding-3-small
export CORTEX_EMBEDDING_DIMS=1536

Provider selection

export CORTEX_EMBEDDING_PROVIDER=<empty> | cohere | ollama

The "any URL containing :11434" detection is a convenience — if you set CORTEX_EMBEDDING_URL=http://localhost:11434 without explicitly setting the provider, you get Ollama mode automatically.

When the API key is missing

If no OPENAI_API_KEY / LLM_API_KEY / COHERE_API_KEY is set and Ollama isn't detected, the binary falls back to MockEmbeddingService (random 384-d vectors) and logs:

warn  No OPENAI_API_KEY or LLM_API_KEY set -- falling back to mock embeddings.

This is a development convenience and produces meaningless recall. Always check startup logs to confirm you're not on mock.

The full env-var surface

Choosing a model

The published LongMemEval-S configuration uses text-embedding-3-small not because it's the best, but because the +0.4pp from large didn't justify the 3× cost — we wanted the published config to be the one we'd recommend to most users. If you have an unusual budget, swap to large.

Batch and retry tuning

The embedding service collects pending requests and packs them into provider calls of up to CORTEX_EMBEDDING_MAX_BATCH_ITEMS items. Larger batches reduce per-request overhead and improve throughput, but increase the latency for the first request in a batch.

OpenAI's text-embedding-3-* models accept up to 2048 inputs per call. Setting CORTEX_EMBEDDING_MAX_BATCH_ITEMS higher than that doesn't error — the client splits transparently — but you stop getting per-batch overhead amortization.

Local-first with Ollama

For development without an API key:

# Install Ollama and pull the model
ollama pull nomic-embed-text

# Point CortexDB at it
export CORTEX_EMBEDDING_URL=http://localhost:11434
export CORTEX_EMBEDDING_MODEL=nomic-embed-text
export CORTEX_EMBEDDING_DIMS=768

# cortex.toml — match the dim
[engine]
vector_dimensions = 768

Ollama runs on the same machine as CortexDB. Expect ~30 ms / embedding on a modern CPU, much faster on a GPU. Throughput is the limit, not latency.

Caching

All HTTP embedding services are wrapped in LruEmbeddingCache, an in-process LRU keyed by (model, text_hash). The cache survives the process lifetime; it's lost on restart.

The cache size isn't currently env-configurable — it sits at the compiled default (~10 K entries). For workloads that re-embed the same text repeatedly (notably reruns of the same eval set), the cache is highly effective; for cold ingest workloads it's near-useless.

Diagnostics

On startup, the binary logs which embedding service it picked:

info  Using HTTP embedding service model=text-embedding-3-small dims=1536
info  Using Ollama local embedding service model=nomic-embed-text dims=768 url=http://localhost:11434
info  Using Cohere embedding service model=embed-multilingual-v3.0 dims=1024
warn  No OPENAI_API_KEY or LLM_API_KEY set -- falling back to mock embeddings.

Always grep for "embedding service" in your startup logs after changing config. Silent recall failures are usually a missing API key or a dim mismatch.

Next steps

• LLM & Answer Generation — the entity-extraction and answer-generation LLMs
• Recall Tuning — how embeddings feed into the recall pipeline
• Storage & Cluster — the HNSW index that consumes these vectors