LLM & Answer Generation

CortexDB calls LLMs in four distinct places, each independently configurable. Mixing them is the norm — e.g. GPT-4o-mini for cheap entity extraction on every write, a strong model for the answer the user sees, an optional verifier on top.

The four LLM call sites

Each has its own model, URL, API key, and disable switch — route them to entirely different providers if you need to. The image bakes a URL and model for the extraction (gpt-4o-mini), verifier (gpt-4.1) and answer (anthropic / claude-opus-4-6) lanes, so a bare provider key switches a lane on: OPENAI_API_KEY alone enables extraction and the verifier, and ANTHROPIC_API_KEY alone enables /v1/answer on claude-opus-4-6.

1. Entity extraction LLM

Runs on /v1/experience ingest to seed the knowledge graph. If disabled, fact/belief layers degrade to text-only matching.

Change it for benchmarks (CORTEX_LLM_DISABLE=1 for pure-text recall), systematic extraction errors (CORTEX_LLM_MODEL=gpt-4o, ~10× cost), or local extraction (CORTEX_LLM_URL=http://localhost:11434/v1 + a local model).

2. Async enrichment LLM (optional)

A heavyweight pipeline doing deeper KG enrichment (multi-hop fact linking, cross-session entity disambiguation). Off by default — it needs the model, an endpoint key and CORTEX_ENRICHMENT_DELAY_SECONDS; without the delay variable the boot log says enrichment OFF and Facts/Beliefs stay empty even with the model and key set.

export CORTEX_ENRICHMENT_MODEL=gpt-4o-mini
export CORTEX_ENRICHMENT_URL=https://api.openai.com/v1
export CORTEX_ENRICHMENT_API_KEY=sk-...
export CORTEX_ENRICHMENT_DELAY_SECONDS=5

gpt-4o-mini is the recommended enrichment model: the cheapest reliable extraction model, about $0.40 per 1,000 events. See Cost Planning.

3. Answer generation LLM

Used by POST /v1/answer to turn a recall pack into a cited natural-language answer.

Recommended (September 2026): gpt-5.6-terra for answers (about 2 to 4 cents each), gpt-5.6-luna for high-volume or simple questions (about 0.2 to 0.4 cents), and gpt-5.6-sol or gpt-6-astra only where answer quality is the product. Terra, Luna and Sol all work as CORTEX_ANSWER_MODEL with CORTEX_ANSWER_PROVIDER=openai. For premium answers use Sol: gpt-6-astra expects max_completion_tokens rather than the max_tokens CortexDB sends, so it isn't usable on the answer lane yet. Costs, monthly estimates and the price-book caveat are on Cost Planning.

Earlier model choices (internal A/B deltas on a 150-question slice — directional, not definitive; the GPT-5.6 models have no published delta on this slice):

Provider notes. Anthropic: set CORTEX_ANSWER_URL to a Bedrock/Vertex endpoint to route hosted Claude. OpenAI: CORTEX_ANSWER_PROVIDER=openai, CORTEX_ANSWER_MODEL=gpt-5.6-terra. Bedrock/Google: set CORTEX_ANSWER_URL to the regional endpoint; auth via the provider's native env vars. Ollama: CORTEX_ANSWER_PROVIDER=ollama, CORTEX_ANSWER_URL=http://localhost:11434, CORTEX_ANSWER_MODEL=qwen2.5:14b (~3–15 s / answer).

4. Verifier LLM (on whenever an OpenAI key is present)

A second LLM call that critiques the answer for hallucination against the citation pack. Doubles latency and cost when enabled. The image bakes CORTEX_VERIFIER_URL=https://api.openai.com/v1 and CORTEX_VERIFIER_MODEL=gpt-4.1, so it switches on as soon as OPENAI_API_KEY (or CORTEX_VERIFIER_API_KEY) is set; set CORTEX_VERIFIER_URL= (empty) to keep it off.

Configure the lane to use it:

export CORTEX_VERIFIER_URL=https://api.openai.com/v1
export CORTEX_VERIFIER_API_KEY=$OPENAI_API_KEY
export CORTEX_VERIFIER_MODEL=gpt-4.1

Using a different family from the answer model avoids sharing the generator's blind spots.

The fallback chain (in cortex.toml)

The [llm] section declares a fallback chain any LLM call can use when the primary provider is unreachable:

[llm]
provider = "openai"
endpoint = ""                      # empty → resolve from provider default
api_key = ""                       # empty → resolve from OPENAI_API_KEY env
model = "gpt-4o-mini"

fallback_provider = "anthropic"
fallback_endpoint = ""             # empty → https://api.anthropic.com/v1
fallback_api_key = ""              # empty → ANTHROPIC_API_KEY env
fallback_model = "claude-sonnet-4-6"

fallback_chain = ["openai", "anthropic", "google"]   # tried in order on cascading failures

max_extraction_batch_size = 8      # entity-extraction items per LLM call
extraction_timeout_ms = 30000      # per-call timeout

Resolution: TOML api_key wins, else the env var for the named provider (OPENAI_API_KEY/ANTHROPIC_API_KEY/LLM_API_KEY); same for endpoint. Call-site CORTEX_* overrides take precedence over the TOML defaults — the TOML chain is a floor, the env vars a ceiling.

Cost ladder

Estimated monthly model spend for an agent writing 10K events a day (300K a month) and answering 1K questions a day (30K a month), using the per-event and per-answer figures on Cost Planning (enrichment about $0.40 per 1,000 events; answers at the rates listed there; OpenAI list prices, September 2026).

Question volume drives the cost, not corpus size: choosing the answer model (and routing simple questions to Luna) is the largest cost lever. The previous version of this table, inherited from the v1 docs, was internally inconsistent on the answer column.

Diagnostics

{
  "answer": "...",
  "citations": [ "..." ],
  "diagnostics": { "answer_model": "gpt-5.6-terra", "recall_ms": 42, "llm_ms": 830, "...": "..." }
}

Without a per-request override, if diagnostics.answer_model doesn't match what you set: a typo made the binary fall back to the compiled default; the fallback chain triggered (primary provider unreachable); or a scope-level policy override is forcing a different model — check /v1/policy/effective?actor=….

Next steps

• Embeddings — the vector side of the inference bill
• Recall Tuning — feed the answer model better context
• Profiles & Presets — the Cost-Optimized profile for an all-mini deployment
• Cost Planning — recommended models and what they cost