openclaw/skills
crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
Installation
$
npx skills add https://github.com/openclaw/skills
Similar popular skills
Related neighbors and high-traction skills in the same topics — useful to compare before installing.
web-crawler
Similar name
starchild-ai-agent/official-skills
Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn, Reddit, Threads, plus robu…
3.9K installs
geo-crawlers
Similar name
zubair-trabzada/geo-seo-claude
AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which A…
583 installs
site-crawler
Similar name
mindmorass/reflex
Crawl and extract content from websites
352 installs
gzh-search-crawler
Similar name
redfox-data/redfox-community
311 installs
xiaohongshu-crawler
Similar name
redfox-data/redfox-community
小红书作品爬取工?
306 installs
douyin-works-crawler
Similar name
redfox-data/redfox-community
抖音作品爬取工?
301 installsAlso in this package
Other skills from openclaw/skills · top by installs.
npx skills add https://github.com/openclaw/skills
More details
Agent compatibility
Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.
Claude Code
Not declared
Cursor
Not declared
Codex
Not declared
GitHub Copilot
Not declared
Windsurf
Not declared
Gemini CLI
Not declared
Cline
Not declared
OpenCode
Not declared
Repository health
Stars
4.5K
Status
Active
History
- First seen on skills.sh
- First recorded snapshot · 1 installs