Bulk extract content from entire websites or site sections with depth and path filtering.
Crawls pages following links up to configurable depth limits and page counts, with path inclusion/exclusion filters to scope extraction Supports async job polling or synchronous waiting with progress display via --wait and --progress flags Offers concurrency control, request delays, and JSON output formatting for integration into agent workflows Part of a four-step escalation pattern: search → scrape → map → crawl, used when single-page extraction is insufficient
Similar popular skills
Related neighbors and high-traction skills in the same topics — useful to compare before installing.
This skill enables automated, bulk extraction of website content using the Firecrawl CLI. It supports depth management, URL filtering, and structured output to local files.
Files included with this skill beyond the listing page.
skill mdSKILL.md1,876 B
docsSUMMARY.md149 B
History
First seen on skills.sh
First recorded snapshot · 79,118 installs
SKILL.md
firecrawl crawl
Bulk extract content from a website. Crawls pages following links up to a depth/limit.
Prerequisite:crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.
Quick start
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json
# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json
# Check status of a running crawl
firecrawl crawl <job-id>
Run firecrawl crawl --help for the full option list.
Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.
Tips
Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).
