Verify every factual claim in an investor deck against a data room of SEC filings, and cite the exact source for each verdict. Two things make that hard, and both are unusual: the numbers that matter live inside tables, and most discrepancies are not lies — they are the same quantity measured on a different accounting basis.
Blue boxes are the only places a model runs. Everything below the fourth rule is deterministic Python — which is why an open-weight model is sufficient, and why the verdict layer is testable with no API key.
Each filing converts to Markdown — tables kept as grids, heading ids kept for
citation — then becomes a tree of section nodes holding titles and spans but no
text. Cached by sha256 of the file, so it is built once ever.
The deck goes in whole — it is short enough to read — and returns typed claims with periods inherited from section headings. Superlatives, projections and team claims are discarded here and never enter the pipeline.
1 model callAll claims become one query; the corpus is the environment. The agent reads every section summary at once, then opens only the nodes it needs. Claims sharing evidence share the fetch.
3 tools · ≤8 turnsThe agent returns figures and node references — never a verdict. Python normalises units, scales and periods, computes every rate and margin, applies the rounding band, then assigns a verdict.
73 tests · no API keyThis is the whole argument, and it is visible in four lines. Snowflake's FY2024 income statement, as it reaches the model under each approach:
── structure-navigating retrieval ────────────────────────── | (in thousands) | FY2024 | FY2023 | | Total revenue | 2,806,489 | 2,065,659 | | Cost of revenue | 898,558 | 717,540 | ── chunk at 512 tokens, embed, retrieve top-k ───────────── chunk 41 "... Total revenue Cost of revenue Gross profit ..." ← labels, no numbers chunk 42 "... 2,806,489 2,065,659 898,558 717,540 ..." ← numbers, no labels, no years
Retrieve chunk 42 and the model sees digits with no idea which is revenue and which is FY2023. It will still answer, confidently. That is the failure mode — not silence, but a wrong number that looks right. Nothing here is chunked, so the question never arises.
| Tool | Returns | Why it exists |
|---|---|---|
| list_documents() | doc ids, types, periods covered | Establishes whether any document covers the claim's period at all — the difference between "looked and found nothing" and "there was nothing to look in". |
| get_structure(doc) | the full tree — titles, summaries, addresses, no text | Small enough to reason over whole, so the agent considers every section before opening one. |
| get_content(doc, [ids]) | markdown with tables intact, plus a resolved citation URL | Batched — one call fetches every node selected. The citation travels with the content. |
| search(query) | BM25 candidates across the corpus | Entity claims have no section to navigate to. Keyword matching beats embeddings on exact proper nouns. |
A 200-page filing indexes to 128 nodes. The agent reads all 128 descriptions and opens two or three sections. That ratio is the economics of the whole design.
| Per verification run | Naive, per claim | This design |
|---|---|---|
| Extract claims from deck | 22 | 1 |
| Locate evidence | 44 | 1 |
| Read sections | 44 | ~8 |
| Compute verdicts | 22 | 0 · Python |
| Total model calls | 132 | ~10 |
Node summaries cost model calls once per document and are then cached forever against the file's content hash, so a re-run over an unchanged data room pays nothing for indexing.
No model, no prompt — an ordered function. Order matters: unit, period and scale are checked
before the numeric band, or $4.2M and $4.2B compare equal
and a thousand-fold error slips through.
(a−b)/b, or a/bRule 5 is the one worth arguing about. A deck reporting ARR against a filing reporting GAAP revenue is not lying — it is measuring differently. Calling that a contradiction is the naive failure, and the fastest way for an analyst to stop trusting the tool. It is detected by running the same tolerance check against the other bases present in the evidence: if the claimed figure matches some correct number, they measured differently rather than lied.
| Ceiling | What happens | Where it changes |
|---|---|---|
| 10,000 documents | The corpus index no longer fits in one prompt, so document selection has to become a retrieval problem of its own. | list_documents |
| Tree too large | Above roughly 500 nodes, all-summaries-at-once stops fitting and selection must descend level by level — losing the ability to pick two nodes in different Items, which is what basis detection depends on. | get_structure |
| Cross-document entities | "Acme Corp" and "Acme Analytics Inc." are not resolved to the same entity. BM25 matches strings, not companies. | search |
| Partial edits | One changed page rebuilds the whole document tree; the cache key is the file hash, not the section. | cache key |
| Single-node cache | Tree cache and run state are one process each. | cache.py · state.py |
| Agent turns | Bounded at 8, and a group that exceeds it fails loudly rather than costing unbounded money. The bound is a guess, not a measurement. | MAX_TURNS |
| Document formats | Text filings only. Scanned pages, image-only charts and spreadsheets yield nothing, and slides whose figures live in chart images are counted and reported rather than silently skipped. | loader.py |
Python · FastAPI · SSE · rank_bm25 · Redis for the tree cache · MongoDB for run
state. Models are open-weight only, reached over an OpenAI-compatible endpoint, so the
provider is one environment variable — Groq, OpenRouter, vLLM or a local Ollama all work
unchanged. No OpenAI, no Anthropic.