xberg-io/crawlberg

1

api Use crawlberg api reference documentation.

xberg-io/crawlberg

1

2

automating-the-browser Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or …

xberg-io/crawlberg

1

3

cli Use crawlberg cli reference documentation.

xberg-io/crawlberg

1

4

crawlberg Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user w…

xberg-io/crawlberg

1

5

crawling-a-site Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlbe…

xberg-io/crawlberg

1

6

headless-fallback Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|…

xberg-io/crawlberg

1

7

mapping-urls Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or…

xberg-io/crawlberg

1

8

mcp Use crawlberg mcp reference documentation.

xberg-io/crawlberg

1

9

scraping-html-to-markdown Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape U…

xberg-io/crawlberg

1

10

serving-the-api Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP ser…

xberg-io/crawlberg

1