LLM & Answer Generation

CortexDB calls LLMs in four distinct places, each independently configurable. Mixing them is the norm — e.g. GPT-4o-mini for cheap entity extraction on every write, a strong model for the answer the user sees, an optional verifier on top.

The four LLM call sites

Each has its own model, URL, API key, and disable switch — route them to entirely different providers if you need to.

1. Entity extraction LLM

Runs on /v1/experience ingest to seed the knowledge graph. If disabled, fact/belief layers degrade to text-only matching.

Change it for benchmarks (CORTEX_LLM_DISABLE=1 for pure-text recall), systematic extraction errors (CORTEX_LLM_MODEL=gpt-4o, ~10× cost), or local extraction (CORTEX_LLM_URL=http://localhost:11434/v1 + a local model).

2. Async enrichment LLM (optional)

A heavyweight pipeline doing deeper KG enrichment (multi-hop fact linking, cross-session entity disambiguation). Off by default — set the model env var to turn it on.

export CORTEX_ENRICHMENT_MODEL=gpt-4o
export CORTEX_ENRICHMENT_URL=https://api.openai.com/v1

3. Answer generation LLM

Used by POST /v1/answer to turn a recall pack into a cited natural-language answer.

Model choices (internal A/B deltas on a 150-question slice — directional, not definitive):

Provider notes. Anthropic: set CORTEX_ANSWER_URL to a Bedrock/Vertex endpoint to route hosted Claude. OpenAI: CORTEX_ANSWER_PROVIDER=openai, CORTEX_ANSWER_MODEL=gpt-4o. Bedrock/Google: set CORTEX_ANSWER_URL to the regional endpoint; auth via the provider's native env vars. Ollama: CORTEX_ANSWER_PROVIDER=ollama, CORTEX_ANSWER_URL=http://localhost:11434, CORTEX_ANSWER_MODEL=qwen2.5:14b (~3–15 s / answer).

4. Verifier LLM (optional, off by default)

A second LLM call that critiques the answer for hallucination against the citation pack. Doubles latency and cost when enabled.

Configure the lane to use it:

export CORTEX_VERIFIER_URL=https://api.openai.com/v1
export CORTEX_VERIFIER_API_KEY=$OPENAI_API_KEY
export CORTEX_VERIFIER_MODEL=gpt-4.1

Using a different family from the answer model avoids sharing the generator's blind spots.

The fallback chain (in cortex.toml)

The [llm] section declares a fallback chain any LLM call can use when the primary provider is unreachable:

[llm]
provider = "openai"
endpoint = ""                      # empty → resolve from provider default
api_key = ""                       # empty → resolve from OPENAI_API_KEY env
model = "gpt-4o-mini"

fallback_provider = "anthropic"
fallback_endpoint = ""             # empty → https://api.anthropic.com/v1
fallback_api_key = ""              # empty → ANTHROPIC_API_KEY env
fallback_model = "claude-sonnet-4-6"

fallback_chain = ["openai", "anthropic", "google"]   # tried in order on cascading failures

max_extraction_batch_size = 8      # entity-extraction items per LLM call
extraction_timeout_ms = 30000      # per-call timeout

Resolution: TOML api_key wins, else the env var for the named provider (OPENAI_API_KEY/ANTHROPIC_API_KEY/LLM_API_KEY); same for endpoint. Call-site CORTEX_* overrides take precedence over the TOML defaults — the TOML chain is a floor, the env vars a ceiling.

Cost ladder

Sample monthly cost for an agent writing 10K events/day, answering 1K queries/day:

The answer call dominates; cutting it is the largest cost lever.

Diagnostics

{
  "answer": "...",
  "citations": [ "..." ],
  "diagnostics": { "answer_model": "gpt-4o", "recall_ms": 42, "llm_ms": 830, "...": "..." }
}

If diagnostics.answer_model doesn't match what you set: a typo made the binary fall back to the compiled default; the fallback chain triggered (primary provider unreachable); or a scope-level policy override is forcing a different model — check /v1/policy/effective?actor=….

Next steps

• Embeddings — the vector side of the inference bill
• Recall Tuning — feed the answer model better context
• Profiles & Presets — the Cost-Optimized profile for an all-mini deployment