dig.rest — how the causal engine works

Shareable architecture overview — concepts and behavior, no source. 2026-07-21.

dig.rest turns the news firehose into a causal graph you can interrogate. Events become nodes; directed, typed cause→effect links ("turns") connect them; every product surface — chains, stories, forecasts, the visual views — is a read over that graph. Everything runs self-hosted on one GPU workstation for $0/month in services.

1. The loop — how a node is born and analyzed

A single always-on worker drives a continuous pipeline; the GPU never idles:

1. Ingest — free public firehoses (GDELT's 15-minute global files, RSS), plus licensed-clean research and primary sources (open-access papers, federal registers, SEC filings, curated public-domain reports). No paid APIs anywhere.
2. Dedup — a deliberately high-precision near-duplicate test. The hard-won rule: token overlap cannot distinguish "same event re-reported" from "distinct reaction to the same event," so anything destructive is adjudicated semantically by the model, never by string similarity.
3. Extract & enrich — full-text extraction, named-entity recognition, model-assigned themes and tone, normalized entity keys. Each event gets a short stable code.
4. Persist — one row per event in Postgres; timestamps carry their true precision (exact / quarter-hour / day) and are never rewritten once stored.
5. Embed — background semantic embeddings (vector index) power search, candidate discovery, and a second dedup gate.
6. Reason ("reedge") — the 9B model, served at up to 1M context, receives batches of candidate pairs — only events that share normalized keys and respect stored time order — and emits typed turns with a why, an evidence quote, a confidence, and a basis (stated in source vs model-inferred).
7. Judge & eval — a calibrated judge model grades a sample of fresh turns nightly; a separate verified tier re-checks stated links; a frozen-fixture eval gate must pass before any prompt or model change ships.

News and research each get a fair share of the GPU via interleaved scheduling; fetch I/O lives on separate threads so the card only ever waits on itself.

2. The graph and its laws

• Turns are typed: Cause · Escalation · Response · Continuation · Reversal (+ plain "follows"), each directional, each carrying evidence and confidence.
• The graph is a DAG by database law, not by model politeness. The edge INSERT itself refuses any link that violates strict stored time-order precedence — in-memory dates drift (the same URL re-arrives from different feeds with different dates), so only the stored row is trusted. Cycles at ~12M nodes: 0, verified nightly.
• Echo chains are refused at the same chokepoint. Same-day "follows" between near-identical titles (one event, many outlets) are rejected by the write path using the exact same predicate the nightly quality metric measures — guard and metric are one function, so they can never diverge. The metric reads 0 today.
• Paper-cites-paper stays in a separate citations store: scholarship is not causality, and keeping it out preserves the eval and the DAG's meaning.

3. The fractal runtime — agents all the way down, safely

The platform's autonomous work runs under a small doctrine ("fractal-agentic") rather than ad-hoc scripts:

• Capsules — every micro-agent job (a resolver tick, a deepen pass) is wrapped in a typed envelope: declared inputs, allowed effects, a time-to-live, and a lineage id.
• Kernel & supervisor — root executions spawn bounded children; deadlines are enforced with kills, spawn storms are broken by a breaker, and descendant counts are capped. Pause and safe-mode switches persist in the database; systemd remains the hard floor beneath everything.
• Effect receipts — irreversible actions (locking a public forecast, sending email) emit receipts marked certain/uncertain, so the system can account for what it did.
• Journal & reconstruction — every execution journals to Postgres and its whole tree can be reconstructed and inspected via an API after the fact.
• Declarations vs reality — a machine-readable map declares each agent's state and allowed effects; a conformance test suite in the deploy gate fails the ship if the declaration and the code disagree. It caught its first real drift on day one.

The same philosophy governs how the platform is built: an orchestrator session specs work, background builder agents implement it, findings are adversarially cross-checked, and every ship passes automated gates (test suite + a headless-browser layout gate).

4. Self-regulation

• Steward — a five-minute cron enforcing standing rules: GPU always working, thermal ceilings, news/research balance, quality KPIs; it can self-heal a stalled worker on a budget and escalates by log line, not by silence.
• Nightly eval row — cycles, echo count, orphan rate, judged precision — the trend is the health signal, and regressions get root-caused against real accrual dates.
• Watchers — every saved dig maintains a Turn Index series (escalation-weighted causal activity with decay), heat z-scores against its own baseline, and tone.
• Guard rails as write-path physics: time-order, echo refusal, per-edge savepoints, schema integrity checks — the model is never trusted to be careful; the database is.

5. What the user gets (features)

• Dig anything — a topic becomes a workspace in seconds, index-served from the corpus with a live-ingest fallback for the long tail.
• THE CHAIN — the strongest cause→effect path through the story, every hop typed, evidenced, and dated; stated links drawn solid, inferred dashed.
• Story & Findings — a stitched, citation-numbered brief where every reference is a real node; findings split news vs research.
• Visual views — Orbit (activity clock + connection chord + family lanes), Web (the dig at the center of a radial constellation with neighbor-topic satellites), Flow (the story as a spine: numbered beats, junction type-chips, tributaries), Tree, Folder with time-sort, Relationships, a 30-day Ticker.
• THE CALL — the house's falsifiable forecast per story ("keeps going / escalates / stalls by <date>", probability locked at make-time), graded by the graph on resolve; its Brier record is public. Read-only by design.
• Background layer — evergreen citable research under every vertical (open-access papers, government primary sources), cross-linked to news by shared keys.
• Ask / Deepen / Neutralize — closed-world Q&A over the dig, on-demand deep research that mints new turns now, and a de-spin rewrite of any event.
• Accounts — passwordless email-code sign-in; saved digs, notes, watchers.

6. The platform (all of it, $0/month)

• One box: a single RTX-class 32 GB workstation. The 9B model serves FP8 at up to 1M context; a smaller judge shares the card. Power-capped on purpose — throughput gains come from killing idle and better batching, not watts.
• Postgres + pgvector as the only datastore (12M nodes, ~1M turns, embeddings, journal, accounts). Nightly dumps to a bulk SSD, retained a week.
• Static frontend on a free CDN tier; the box is never publicly exposed — the API reaches the world through an outbound tunnel only.
• Free data in, open-weight model, permissive licenses only — AGPL and non-commercial weights are launch-blockers by policy.

Live snapshot (measured 2026-07-21)