Retention & WAL Maintenance

Operator-driven expiry of old events and reclamation of dead write-ahead-log rows (v0.9.10+) — per-scope rules, dry-run sweeps, and the WAL census.

From v0.9.10 the retention and compaction of the production write-ahead log are driven by operator routes rather than a background policy you cannot see. Every route below was verified live on v0.9.13. Expiry is irreversible erasure, so every mutating call defaults to dry_run: true and you opt in to apply.

Retention is a delete path

A retention sweep that runs with dry_run: false erases expired events from the WAL and, when CORTEX_RETENTION_COMPACT_AFTER_SWEEP is on (default), reclaims their rows straight away. It is the same class of operation as erasures, driven by age instead of by selector. Take a verified backup before the first non-dry run.

Retention rules

A rule is stored per scope and applies to that scope's events. Age is measured from when the event was recorded, not from its observed_at (verified: an event with observed_at in 2025 written today was scanned and not expired under a one-day rule).

PUT /v1/admin/retention/policies
{ "scope": "org:acme", "default_retention_days": 30, "per_source_days": { "source:slack": 7 } }
→ 200 { "scope": "org:acme", "policy": { … }, "cursor_reset": true }

GET    /v1/admin/retention/policies?scope=org:acme   → 200 { scope, policy, effective_days }  |  404 NOT_FOUND
DELETE /v1/admin/retention/policies?scope=org:acme   → 200  |  404 NOT_FOUND

default_retention_days applies to every event in the scope. per_source_days overrides it per connector; keys must be source:<system> labels (the label connectors stamp on their events). A bare key such as "slack" is refused: 422 INVALID_RETENTION_POLICY: per-source key "slack" is not a source:<system> label.
The body is strict: an unknown field (for example max_age_days) is a 422.
Storing a rule resets the sweep cursor for that scope (cursor_reset: true) so the next tick rescans it.

The sweep

POST /v1/admin/retention/sweep
{ "dry_run": true }            # the default — omit the body for a dry run
→ 200 { dry_run, rescan, trigger: "api", cursor_before, cursor_after, watermark, scanned, expired,
        erased, kept_forever, held: { legal_hold, unindexed }, scopes: [...], batches,
        stopped: "no_policy" | "youngest_reached" | …, errors: [], compaction, elapsed_ms }

stopped tells you why the tick ended: no_policy when no scope has a rule, youngest_reached when the scan reached events newer than any rule. held counts rows that were due but kept: under a legal hold, or not yet indexed (an event still captured is never expired). Pass { "dry_run": false } to apply; the response shape is identical, with erased and compaction filled in.

A scheduler also runs a tick every CORTEX_RETENTION_SWEEP_INTERVAL_SECS (default 3600); with no rules stored a tick stops immediately with no_policy. Set the interval to 0 for operator-triggered sweeps only.

GET /v1/admin/retention
→ 200 { enabled, scheduler_disabled, deployment_default_days, sweep_interval_secs,
        max_events_per_tick, tick_budget_secs, compact_after_sweep, policies: [...],
        cursor, watermark, running, last, totals: { ticks, scanned, expired, erased, held, errors, … },
        wal: { live_count, … } }

The WAL census and compaction

GET /v1/admin/wal
→ 200 { live_count, next_offset, head_offset, captured_head_offset, partitions,
        consumer_offsets: { indexer: n, … }, retired_marker_count, native_profile_blocked, backend: "unified" }

POST /v1/admin/wal/compact
{ "dry_run": true }            # the default
→ 200 { scanned, removed_tombstones, removed_orphaned_twins, skipped_in_use, skipped_above_bound,
        bytes_removed, live_count_before, live_count_after, ranges_compacted, dry_run,
        truncated_by_budget, elapsed_ms }

Compaction only reclaims rows that are already dead (tombstoned by a forget, an erasure or a retention sweep). It never expires anything itself, and it skips rows a consumer still needs (skipped_in_use) or that sit above the consumer floor (skipped_above_bound). v0.9.10 also batched the full-text-index commit during large erasures; a 10,000-document erasure previously meant roughly ten minutes of fsync with ingest blocked.

Configuration

All defaults verified from the boot config_lint dump on v0.9.13.

Env var	Default	What it controls
`CORTEX_RETENTION_DEFAULT_DAYS`	unset	Deployment-wide default when a scope has no rule (falls back to `cortex.toml` `[governance] default_retention_ttl_secs`).
`CORTEX_RETENTION_SWEEP_INTERVAL_SECS`	`3600`	Scheduler period. `0` = operator-triggered sweeps only; minimum 60 when non-zero.
`CORTEX_RETENTION_MAX_EVENTS_PER_TICK`	`10000`	Expired rows decided per tick (legally held rows included).
`CORTEX_RETENTION_TICK_BUDGET_SECS`	`300`	Wall-clock ceiling on one tick's scan.
`CORTEX_RETENTION_COMPACT_AFTER_SWEEP`	`true`	Compact dead rows after a tick that erased anything.

CLI and SDK

cortexdb-cli 0.5.4 exposes the same surface under cortexdb admin retention (show, set, sweep, delete) and cortexdb admin wal (status, compact); they are sub-groups of admin, not top-level commands. The Python SDK (cortexdbai 0.11.2, sync client only) has retention_status / retention_policy_get / retention_policy_set / retention_policy_delete / retention_sweep and wal_status / wal_compact. See CLI and Python SDK.

Erasures — selector-driven GDPR deletion.
Forget — operational deletion with preview.
Storage & Cluster — the WAL, snapshots and the extraction cache.
Audit Trail — every erasure is recorded.