Ollama
Use a locally-hosted Ollama model as the answer model and entity-extraction LLM for CortexDB.
CortexDB routes its LLM calls through configured routers. Point them at an Ollama instance to keep all inference local — no data leaves your network.
Two lanes: extraction (CORTEX_LLM_*) and answer (CORTEX_ANSWER_*)
CORTEX_LLM_* configures the entity-extraction lane. /v1/answer uses the separate
CORTEX_ANSWER_* lane, which defaults to provider=anthropic / claude-opus-4-6 (baked into the
image). Setting CORTEX_LLM_* alone leaves /v1/answer on the disabled Anthropic default — you
must set both lanes. See Self-hosting defaults.
Deployment configuration
Set these on the cortex server:
# Entity-extraction lane
CORTEX_LLM_URL=http://localhost:11434/v1
CORTEX_LLM_MODEL=llama3.1:70b
CORTEX_LLM_API_KEY=ollama # placeholder; Ollama ignores it
# Answer lane — REQUIRED for /v1/answer to route through Ollama
CORTEX_ANSWER_PROVIDER=ollama
CORTEX_ANSWER_URL=http://localhost:11434
CORTEX_ANSWER_MODEL=llama3.1:70b
CORTEX_ANSWER_API_KEY=ollama # placeholder
# Embedding (for vector recall)
CORTEX_EMBEDDING_PROVIDER=ollama
CORTEX_EMBEDDING_MODEL=nomic-embed-text
CORTEX_EMBEDDING_DIMS=768
CORTEX_EMBEDDING_API_KEY=ollama # non-empty required even for OllamaRestart the service to pick up the new config, then confirm /v1/answer is live:
curl -s http://localhost:3141/v1/admin/ready # degraded:false = real embeddingsPer-request answer-model override
Once the router knows about Ollama, pin individual /v1/answer calls to specific local models:
client.answer(
scope="org:acme/user:alice",
question="What did we decide about the launch?",
answer_model="ollama/llama3.1:70b",
)answer_model accepts any <provider>/<model-name> the configured router supports.
Pulling models
ollama pull llama3.1:70b
ollama pull nomic-embed-textFor benchmark-grade accuracy use the largest model your GPU fits (Llama 3.1 70B or Qwen 2.5 72B); smaller models lose 5–15 pp on multi-session reasoning.