ultradoc — answer questions from the source, not from memory

ultradoc answers ultra-precise questions about an open-source project by retrieving grounded evidence and reasoning over it. The deterministic engine (scripts/ultradoc.mjs, zero-dependency Node, no keys, no npm install) does the searching and indexing with code; your job is to read the retrieved evidence and write a precise, cited answer. This is enforced: ultradoc check fails if any citation does not resolve to retrieved evidence, and re-validates each code/docs excerpt against the pinned clone.

The core rule: do not answer from your own knowledge of the library. Your
training data is stale and hallucinates APIs. Answer only from the
evidence ultradoc retrieves from the actual repo, issues, PRs, docs and web.
If the evidence does not cover it, say so and retrieve more — never guess.

Route the ask

node scripts/ultradoc.mjs <command> — run --help for the full flag surface.

ask persists a run (EVIDENCE.md, evidence.json, meta.json, drill-plan.json) beside the clone at <clone>/.ultradoc/runs/<id> — a stable, commit-pinned home reused across questions — unless you pass --out. Default sources are code,issues,prs,docs; the clone and index are cached per user, so every drill after the first ask is near-free (references/tuning.md).

Budget the run

Match the effort to the ask; escalate on evidence, not on reflex.

The gates do not scale down. check --strict always. verify + check --semantic always, except on the fast path when the answer is ≤3 claims all cited to code/docs excerpts — those are exactly the citations check re-validates line-by-line against the pinned clone. Any claim that interprets evidence, or rests on an issue/PR, goes through verify.

Workflow

You own this task end-to-end: return one grounded, cited answer, never a half-retrieved dossier. Retrieval and verification are independent, near-free calls — parallelize when your harness can (batch independent drills in one message; fan out to subagents or a workflow if available), inline otherwise. See references/orchestration.md.

When a command fails, recover — don't guess:

1. Resolve the target. Identify the project and the precise question. With

only a name, find the canonical repo URL (your WebSearch, or ask if ambiguous). Note any version/branch the user cares about (--ref). Several questions about the same repo, or you don't know where a behavior lives? Run overview once and drill its Core modules first.

1. Retrieve. Derive 2–3 query variants — the engine searches literally,

so phrasing decides what surfaces. Spend variants on synonyms and identifiers (retryBackoff, MAX_RETRIES, the literal error string), never on inflections; conceptual "why designed this way" questions lead with docs/discussions/web or --semantic, not code. The full variant and source tables are in references/retrieval-playbook.md.

Split a multi-part question into sub-questions now; each needs its own evidence or an explicit "unknown" at the end. Then run ask with the best variant and the sources that fit — ultradoc auto-discovers the project's official docs URL (override with --docs-url). The remaining variants × sources are a fan-out, which ask persists as drill-plan.json.

1. Read the dossier. Retrieval notes first — they say what this run could

not reach (a capped index, a regex-tier symbol scan, a sliced call-site list, a rate-limited provider) and therefore bound what you may claim. Then the item titles, then the snippets. references/reading-evidence.md decodes every note, the item labels, meta.symbolSpan/meta.confidence, and what check's warnings mean. Read the real code/issue/PR/doc text — never the file name.

1. Drill the gaps. Fan the variant drills out in parallel from the start;

iterate in rounds only when a fan-out surfaces new leads, and stop per the budget table. Two rules decide quality: triage before writing — cite an item only if its snippet names the symbol/behavior or describes the same mechanism, never on a shared keyword; and re-query instead of re-reading — two off-topic dossiers mean the wording is wrong. A sub-question still unsupported at the cap is an explicit unknown, never filled from memory.

1. Write the answer to ANSWER.md in the run folder, per **The answer

contract** below and references/citation-format.md.

1. Validate (two layers).

- Structural: check --run <dir> --strict fails on any citation that doesn't resolve, on an answer with no citations, and (with --strict) on any uncited claim — so the answer can't be mostly memory around one real reference. Fix and re-run until it passes. - Semantic (adversarial support-check): verify --run <dir> writes a claim↔evidence worklist. Judge each pair as a skeptic: default to unsupported/refuted unless the cited snippet literally backs the claim (supported · partial · refuted · unsupported + a note). A pair flagged ⚠ cross-check is grounded in an issue/PR and must be judged against CURRENT code. Collect every verdict into a single verdicts.json (references/orchestration.md has the verdict table and the return contract), then: `` node scripts/ultradoc.mjs verify --apply verdicts.json --run <dir> node scripts/ultradoc.mjs check --semantic --run <dir> ` check --semantic fails when VERIFY.json is missing (or pass --allow-unverified to skip the gate explicitly), and on any refuted or unsupported claim — closing the gap where a citation resolves but does not back the claim. Fix the claim (re-cite, weaken, drop, or retrieve better) and re-verify. Then self-review against references/answer-rubric.md`.

1. Present per the answer contract. references/worked-example.md walks one

full run — retrieve, triage, write, both gates, hand-off.

The answer contract

ANSWER.md is: a lead line that answers the question — cited like any other claim, --strict counts it; then one claim per sentence, each carrying the evidence id it rests on; identifiers, defaults and values quoted verbatim from the excerpt; then a ## Unknowns section naming what the evidence did not settle (that heading is exempt from the coverage gate — an unknown cites nothing by construction, so it never has to be dropped to get a green run). Put the commit from meta.json in an HTML comment: check ignores comments, and a bare "Verified against abc1234." is an uncited claim that fails --strict.

Your message to the user is: that answer, plus the clickable refs from the evidence (file:line, issue/PR numbers, doc/SO/web URLs), the commit it was verified against, and the unknowns — stated, not filled. Write it in the conversation's language; identifiers, flags and file paths stay verbatim.

Red flags — you are about to answer from memory

Generate a documentation

When the user wants a whole-project (or whole-package) doc rather than one answer, doc is the same grounded loop fanned out over a section outline:

1. Scaffold. doc --repo <url> [--package <p>] [--sources …]. The engine

builds a deterministic outline (overview, install/usage, public API or one section per workspace package, configuration, architecture, then one section per central subsystem), retrieves a dossier per section, merges them into one evidence.json with global [E#] ids, and writes DOC.todo.md + DOC.plan.json under <clone>/.ultradoc/doc/, with an ARCHITECTURE.mmd module diagram — navigation like OVERVIEW.md, never cited.

1. Write each section. Read DOC.todo.md and EVIDENCE.md, then write

DOC.md: one section per outline entry, every claim cited [E#]. A section with thin evidence is a fan-out unit — drill it or mark the gap an explicit unknown; never write from memory.

1. Validate & present. check --run <doc-dir> (and verify +

check --semantic, exactly as in step 6 — both auto-detect DOC.md). Fix until grounded, then present DOC.md pinned to its commit.

Orchestration — route by harness

The per-item work fans out: drill-plan.json (one cell per {query-variant × source}, plus one symbol cell per identifier the question names), VERIFY.todo.json (one claim↔evidence pair) and DOC.plan.json (one section) are independent worklists. orchestrate emits the orchestration from the CURRENT worklists, with absolute paths and real item ids baked in:

node scripts/ultradoc.mjs orchestrate --run <dir> [--phase drill|verify|doc] [--eco] [--list]

Fan-out is an optimization, never a requirement — the gates are harness-independent and every phase has a sequential fallback with identical artifacts. Subagents never write; the folds stay with you, the orchestrator. Re-run orchestrate whenever a worklist changes (emission is deterministic and idempotent); --phase <p> before its worklist exists fails and names the command that produces it.

Optional semantic mode (fully local, no API key)

Tier-1 search (ripgrep + symbol index) is the default and needs nothing. Add --semantic when the question describes what the codebase names differently — "which helper works out how long to wait before trying again" finds computeBackoff, which no lexical wording reaches. Two keyless backends:

• static — semantic pull (~21 MB, once, no container). Embeds symbol names

and signatures: answers which declaration.

• docker — semantic up, then --semantic-tier docker. Embeds the real

content of code and docs, so it is the one that answers why it is designed this way. Reach for it when the question needs prose.

--semantic-tier auto (default) tries endpoint → static → docker; with no backend --semantic names the command that would enable one and falls back to Tier 1. See references/semantic-setup.md.

Optional page extraction (fully local, no API key)

firecrawl up|down|status runs a keyless self-hosted Firecrawl; fetched pages then arrive as browser-rendered main-content markdown instead of regex-stripped HTML. Reach for it when a docs/web excerpt is nav/cookie chrome or a page came back empty (JS-rendered). Never required — it degrades to the stripper, noting any failure. See references/web-discovery.md.

References