Integrations
vLLM
Route CortexDB's answer and extraction LLM calls through a self-hosted vLLM server.
vLLM serves any Hugging Face model behind an OpenAI-compatible endpoint, so CortexDB can keep all inference on your own hardware.
Set both the extraction and answer lanes
CORTEX_LLM_* is the entity-extraction lane; /v1/answer uses the separate CORTEX_ANSWER_*
lane (default anthropic/claude-opus-4-6). Set both. See
Self-hosting defaults.
Start vLLM
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.3-70B-Instruct \
--port 8000 \
--api-key local-secretDeployment configuration
# Extraction lane
CORTEX_LLM_URL=http://localhost:8000/v1
CORTEX_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
CORTEX_LLM_API_KEY=local-secret # matches --api-key (or a placeholder if unset)
# Answer lane (OpenAI-compatible)
CORTEX_ANSWER_PROVIDER=openai
CORTEX_ANSWER_URL=http://localhost:8000/v1
CORTEX_ANSWER_MODEL=meta-llama/Llama-3.3-70B-Instruct
CORTEX_ANSWER_API_KEY=local-secretKeep CORTEX_EMBEDDING_* on a local Ollama or an embedding-serving vLLM instance. See
Embeddings.
Per-request override
client.answer(scope="org:acme/user:alice", question="…",
answer_model="openai/meta-llama/Llama-3.3-70B-Instruct")