Bulk extract content from entire websites or site sections with depth and path filtering.
Crawls pages following links up to configurable depth limits and page counts, with path inclusion/exclusion filters to scope extraction Supports async job polling or synchronous waiting with progress display via --wait and --progress flags Offers concurrency control, request delays, and JSON output formatting for integration into agent workflows Part of a four-step escalation pattern: search → scrape → map → crawl, used when single-page extraction is insufficient
Similar popular skills
Related neighbors and high-traction skills in the same topics — useful to compare before installing.
Files included with this skill beyond the listing page.
skill mdSKILL.md1,876 B
docsSUMMARY.md149 B
History
First seen on skills.sh
First recorded snapshot · 79,118 installs
SKILL.md
firecrawl crawl
Bulk extract content from a website. Crawls pages following links up to a depth/limit.
Prerequisite:crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.
Quick start
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json
# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json
# Check status of a running crawl
firecrawl crawl <job-id>
Run firecrawl crawl --help for the full option list.
Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.
Tips
Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).
