Web Research Workflow
Unified approach for web content research that automatically selects the right tool for each situation.
Quick Decision Tree
URL to research
│
▼
┌─────────────────┐
│ 1. Try WebFetch │ ← Fast, free, no overhead
│ (always try) │
└─────────────────┘
│
Content OK? ──Yes──► Parse and return
│
No (empty/partial/<500 chars)
│
▼
┌───────────────────────┐
│ 2. TAVILY_API_KEY set?│
└───────────────────────┘
│ │
Yes No ──► Skip to step 3
│
▼
┌───────────────────────────┐
│ Tavily search/extract/ │ ← Raw markdown, batch URLs
│ crawl/research │
└───────────────────────────┘
│
Content OK? ──Yes──► Parse and return
│
No (JS-rendered/auth-required)
│
▼
┌─────────────────────┐
│ 3. Use agent-browser │ ← Full browser, last resort
└─────────────────────┘
│
├─ SPA (react/vue/angular) ──► wait --load networkidle
├─ Login required ──► auth flow + state save
├─ Dynamic content ──► wait --text "Expected"
└─ Multi-page ──► crawl pattern
Tavily Enhanced Research
When TAVILYAPIKEY is set, Tavily provides a powerful middle tier between WebFetch and agent-browser. It returns raw markdown content, supports batch URL extraction, and offers semantic search with relevance scoring. If TAVILYAPIKEY is not set, the 3-tier tree collapses to 2-tier (WebFetch → agent-browser) automatically.
Load details: Read("rules/tool-selection.md") for when-to-use-what tables, escalation heuristics, SPA detection patterns, and cost awareness.
Load details: Read("references/tavily-api.md") for Search, Extract, Map, Crawl, and Research endpoint examples and options.
Browser Patterns
For content requiring JavaScript rendering, authentication, or multi-page crawling, fall back to agent-browser.
Load details: Read("rules/browser-patterns.md") for auto-fallback, authentication flow, multi-page research patterns, best practices, and troubleshooting.
Competitive Monitoring
Track competitor websites for changes in pricing, features, positioning, and content.
Load details: Read("rules/monitoring-competitor.md") for snapshot capture, structured data extraction, and change classification.
Load details: Read("rules/monitoring-change-detection.md") for diff detection, structured comparison, Tavily site discovery, and CI automation.
Change Classification
| Severity |
Examples |
Action |
| Critical |
Price increase/decrease, major feature change |
Immediate alert |
| High |
New feature added, feature removed |
Review required |
| Medium |
Copy changes, positioning shift |
Note for analysis |
| Low |
Typos, minor styling |
Log only |
Integration with Agents
This skill is used by:
web-research-analyst - Primary user
market-intelligence - Competitor research
product-strategist - Deep competitive analysis
Related Skills
browser-content-capture - Detailed browser patterns
agent-browser - CLI reference
Version: 1.3.0 (March 2026)