PDF Text Extractor

Triggers & routing

• Trigger: PDF download, fulltext, extract text, papers/pdfs, 全文抽取, 下载PDF.
• Use when: queries.md 设置 evidence_mode: fulltext（或你明确需要全文证据）并希望为 paper notes/claims 提供更强 evidence。

Optionally collect full-text snippets to deepen evidence beyond abstracts.

This skill is intentionally conservative: in many survey runs, abstract/snippet mode is enough and avoids heavy downloads.

Inputs

• papers/coreset.csv (expects paperid, title, and ideally pdfurl/arxivid/url)
• Optional: outline/mapping.tsv (to prioritize mapped papers)

Outputs

• papers/fulltext_index.jsonl (one record per attempted paper)
• Side artifacts:

- papers/pdfs/<paperid>.pdf (cached downloads) - papers/fulltext/<paperid>.txt (extracted text)

Decision: evidence mode

• queries.md can set evidence_mode: "abstract" | "fulltext".

- abstract (default template): do not download; write an index that clearly records skipping. - fulltext: download PDFs (when possible) and extract text to papers/fulltext/.

Local PDFs Mode

When you cannot/should not download PDFs (restricted network, rate limits, no permission), provide PDFs manually and run in “local PDFs only” mode.

• PDF naming convention: papers/pdfs/<paperid>.pdf where <paperid> matches papers/core_set.csv.
• Set - evidence_mode: "fulltext" in queries.md.
• Run: uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only

If PDFs are missing, the script writes a to-do list:

• output/MISSING_PDFS.md (human-readable summary)
• papers/missing_pdfs.csv (machine-readable list)

Workflow (heuristic)

1. Read papers/core_set.csv.
2. If outline/mapping.tsv exists, prioritize mapped papers first.
3. For each selected paper (fulltext mode):

- resolve pdfurl (use pdfurl, else derive from arxivid/url when possible) - download to papers/pdfs/<paperid>.pdf if missing - extract a reasonable prefix of text to papers/fulltext/<paperid>.txt - append/update a JSONL record in papers/fulltextindex.jsonl with status + stats

1. Never overwrite existing extracted text unless explicitly requested (delete the .txt to re-extract).

Quality checklist

• papers/fulltext_index.jsonl exists and is non-empty.
• If evidence_mode: "fulltext": at least a small but non-trivial subset has extracted text (strict mode blocks if extraction coverage is near-zero).
• If evidencemode: "abstract": the index covers every papers/coreset.csv paper and every record clearly reflects skipmodeabstract (no downloads attempted). fulltextmaxpapers does not truncate this zero-download index.

Script

Quick Start

• uv run python .codex/skills/pdf-text-extractor/scripts/run.py --help
• uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace>

All Options

• --max-papers <n>: cap number of papers processed (can be overridden by queries.md)
• --max-pages <n>: extract at most N pages per PDF
• --min-chars <n>: minimum extracted chars to count as OK
• --sleep <sec>: delay between downloads
• --local-pdfs-only: do not download; only use papers/pdfs/<paper_id>.pdf if present
• queries.md supports: evidencemode, fulltextmaxpapers, fulltextmaxpages, fulltextmin_chars

Examples

• Abstract mode (no downloads):

- Set - evidencemode: "abstract" in queries.md, then run the script (it will emit papers/fulltextindex.jsonl with skip statuses)

• Fulltext mode with local PDFs only:

- Set - evidence_mode: "fulltext" in queries.md, put PDFs under papers/pdfs/, then run: uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --local-pdfs-only

• Fulltext mode with smaller budget:

- uv run python .codex/skills/pdf-text-extractor/scripts/run.py --workspace <workspace> --max-papers 20 --max-pages 4 --min-chars 1200

Notes

• Downloads are cached under papers/pdfs/; extracted text is cached under papers/fulltext/.
• The script does not overwrite existing extracted text unless you delete the .txt file.

Troubleshooting

Issue: no PDFs are available to download

Fix:

• Use evidence_mode: abstract (default) or provide local PDFs under papers/pdfs/ and rerun with --local-pdfs-only.

Issue: extracted text is empty/garbled

Fix:

• Try a different extraction backend if supported; otherwise mark the paper as abstract evidence level and avoid strong fulltext claims.