SkillsAdd

openclaw/skills

crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

First seen Apr 4, 2026

Installation

$ npx skills add https://github.com/openclaw/skills
View on skills.sh ↗ GitHub repository ↗
Feature your product here Skill page unit · reach builders installing agent skills View pricing →

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

web-crawler Similar name
starchild-ai-agent/official-skills

Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn, Reddit, Threads, plus robu…

3.9K installs
geo-crawlers Similar name
zubair-trabzada/geo-seo-claude

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which A…

13 installs
site-crawler Similar name
mindmorass/reflex

Crawl and extract content from websites

352 installs
gzh-search-crawler Similar name
redfox-data/redfox-community

公众号搜索爬虫 — 通过关键词搜索微信公众号文章，终端表格展示（标题/作者/阅读/点赞/分享/收藏/发布…

311 installs
xiaohongshu-crawler Similar name
redfox-data/redfox-community

小红书作品爬取工?

306 installs
douyin-works-crawler Similar name
redfox-data/redfox-community

抖音作品爬取工?

301 installs

Also in this package

Other skills from openclaw/skills · top by installs.

npx skills add https://github.com/openclaw/skills
find-skills 159 yahoo-finance 155 mcporter 100 agent-browser 85 skill-vetter 85 proactive-agent 52 github 47 summarize 46 weather 45 gog 43

Browse all from openclaw/skills

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Repository health

Stars 4.5K
Status Active

History

  1. First seen on skills.sh
  2. First recorded snapshot · 1 installs