AI Tech RSS Fetch

Core Goal

• Subscribe to RSS/Atom sources.
• Persist feed and entry metadata to SQLite.
• Deduplicate entries with layered identity keys plus content fingerprints.
• Keep only metadata; do not fetch full article bodies and do not summarize.

Triggering Conditions

• Receive a request to subscribe RSS feeds from URLs or OPML.
• Receive a request to run incremental RSS sync reliably.
• Need stable metadata persistence for downstream processing.
• Need dedupe-safe storage of feed items over repeated runs.

Workflow

1. Prepare runtime and database.

• Ensure dependency is installed: python3 -m pip install feedparser.
• In multi-agent runtimes, pin DB to an absolute path before any command:

export AI_RSS_DB_PATH="/absolute/path/to/workspace-rss-bot/ai_rss.db"

• Initialize SQLite schema once:

python3 scripts/rss_subscribe.py init-db --db "$AI_RSS_DB_PATH"

1. Add feed subscriptions.

• Add one feed URL:

python3 scripts/rss_subscribe.py add-feed --db "$AI_RSS_DB_PATH" --url "https://example.com/feed.xml"

• Import feeds from OPML:

python3 scripts/rss_subscribe.py import-opml --db "$AI_RSS_DB_PATH" --opml assets/hn-popular-blogs-2025.opml

1. Run incremental sync.

• Fetch active feeds and store metadata:

python3 scripts/rss_subscribe.py sync --db "$AI_RSS_DB_PATH" --max-feeds 20 --max-items-per-feed 100

• Optional one-feed sync:

python3 scripts/rss_subscribe.py sync --db "$AI_RSS_DB_PATH" --feed-url "https://example.com/feed.xml"

1. Query persisted metadata.

• List feeds:

python3 scripts/rss_subscribe.py list-feeds --db "$AI_RSS_DB_PATH" --limit 50

• List recent entries:

python3 scripts/rss_subscribe.py list-entries --db "$AI_RSS_DB_PATH" --limit 100

Input Requirements

• Supported inputs:

- RSS XML feed URLs. - OPML feed list files.

Output Contract (Metadata Only)

• Persist feeds metadata to SQLite:

- feedurl, feedtitle, siteurl, etag, lastmodified, status fields.

• Persist entries metadata to SQLite:

- id, dedupekey (compat primary identity snapshot), guid, url, canonicalurl, title, author, publishedat, updatedat, summary, categories, contenthash, matchconfidence, timestamps.

• Persist entry_identities mapping table to SQLite:

- entryid, keytype, keyvalue, createdat. - Supported key types: guid, canonicalurl, legacyguid, fallback_hash.

• Do not store generated summaries and do not create archive markdown files.

Configurable Parameters

• db_path
• AIRSSDB_PATH (recommended absolute path in multi-agent runtime)
• opml_path
• feed_urls
• maxfeedsper_run
• maxitemsper_feed
• user_agent
• seenttldays
• enableconditionalget
• Example config: assets/config.example.json

Error and Boundary Handling

• Feed HTTP/network failure: keep syncing other feeds and record last_error.
• Feed 304 Not Modified: skip entry parsing and keep state.
• Missing guid and link: use hashed fallback identity and set match_confidence=low.
• Dependency missing (feedparser): return install guidance.

Final Output Checklist (Required)

• core goal
• trigger conditions
• input requirements
• metadata schema
• dedupe and sync rules
• command workflow
• configurable parameters
• error handling

Use the following simplified checklist verbatim when the user requests it:

核心目标
输入需求
触发条件
元数据模型
去重与同步规则
命令流程
可配置参数
错误处理

References

• references/input-model.md
• references/output-rules.md
• references/time-range-rules.md

Assets

• assets/hn-popular-blogs-2025.opml (candidate feed pool)
• assets/config.example.json

Scripts

• scripts/rss_subscribe.py