eceee News Fulltext Fetch

Core Goal

• Discover news article URLs from https://www.eceee.org/all-news/.
• Persist discovered entry metadata into SQLite.
• Fetch and extract article body text from each entry page.
• Persist status and text in a companion table (entry_content) with retry-safe updates.

Triggering Conditions

• Receive a request to extract full text from eceee news archive pages.
• Receive a request to run incremental fulltext sync for eceee news links.
• Need a resilient local SQLite queue for discovery + extraction + retries.

Workflow

1. Initialize database.

export ECEEE_NEWS_DB_PATH="/absolute/path/to/eceee_news.db"
python3 scripts/fulltext_fetch.py init-db --db "$ECEEE_NEWS_DB_PATH"

1. Discover links and fetch fulltext incrementally.

python3 scripts/fulltext_fetch.py sync \
  --db "$ECEEE_NEWS_DB_PATH" \
  --index-url "https://www.eceee.org/all-news/" \
  --limit 50 \
  --min-chars 180

1. Discover only (refresh URL catalog without fetching bodies).

python3 scripts/fulltext_fetch.py sync \
  --db "$ECEEE_NEWS_DB_PATH" \
  --discover-only

1. Fetch one entry on demand.

python3 scripts/fulltext_fetch.py fetch-entry \
  --db "$ECEEE_NEWS_DB_PATH" \
  --entry-id 123

Or by URL:

python3 scripts/fulltext_fetch.py fetch-entry \
  --db "$ECEEE_NEWS_DB_PATH" \
  --url "https://www.eceee.org/all-news/news/example-slug/"

1. Inspect stored state.

python3 scripts/fulltext_fetch.py list-entries --db "$ECEEE_NEWS_DB_PATH" --limit 100
python3 scripts/fulltext_fetch.py list-content --db "$ECEEE_NEWS_DB_PATH" --status ready --limit 100

Data Contract

• entries table stores discovery metadata:

- url, title, publishedat - discoveredat, lastseenat

• entrycontent table stores extraction result (one row per entryid):

- sourceurl, finalurl, httpstatus - extractor (trafilatura, html-parser, or none) - contenttext, contenthash, contentlength - status (ready or failed) - retry fields + timestamps

Extraction and Update Rules

• Discovery source is https://www.eceee.org/all-news/, extracting anchor tags with class newslink under /all-news/news/.
• Fulltext extraction uses article main content region (mainContentColumn) and removes related-news/share blocks.
• Extraction path:

1. trafilatura (if installed and not disabled) 2. built-in HTML parser fallback

• Upsert by entry_id:

- Success: set ready, write text/hash/length, reset retry counters. - Failure with existing ready content: keep old content, update error/retry metadata. - Failure without ready content: set failed, increment retries, set nextretryat.

Configurable Parameters

• --db
• ECEEENEWSDB_PATH
• --index-url
• --discover-only
• --limit
• --force
• --only-failed
• --since-date
• --refetch-days
• --oldest-first
• --timeout
• --max-bytes
• --min-chars
• --max-retries
• --retry-backoff-minutes
• --user-agent
• --disable-trafilatura
• --fail-on-errors

Error Handling

• Index fetch/parse failure returns actionable error.
• HTTP/network/content-type failures are recorded per entry and do not stop the whole sync batch.
• Short extracted text (< --min-chars) is treated as failed to avoid low-quality bodies.
• Retry queue is controlled via max_retries + exponential backoff.

References

• references/schema.md
• references/fetch-rules.md

Assets

• assets/config.example.json

Scripts

• scripts/fulltext_fetch.py