SKILL.md
You are adding an empty web-poet page object to a Scrapy project.
Read ${CLAUDESKILLDIR}/../scrape/references/python-environments.md.
Input
The raw argument string is $ARGUMENTS. Split it into up to 6 whitespace-separated positional arguments:
- filepath: path to the .py file to create or append to (e.g.
booksproject/pages/bookstoscrapecom.py) - class_name: page object class name (e.g.
ProductPage) - domain: domain for
@handle_urls(e.g.books.toscrape.com) - baseclass: base class import path (e.g.
webpoet.WebPage) - itemclass: item class import path (e.g.
booksproject.items.ProductItem) - fields: optional, comma-separated field names (e.g.
name,price,rating)
Process
Run from the project root (the directory containing pyproject.toml) so the item class is importable for auto-detecting required fields:
uv run --project . --with libcst ${CLAUDE_SKILL_DIR}/scripts/add_page_object.py \
FILE_PATH CLASS_NAME DOMAIN BASE_CLASS ITEM_CLASSRequired fields (those with no default in the item class) are detected automatically via itemadapter and get @field stubs. If all fields have defaults the class body is pass. Pass --fields name,price to override auto-detection.
The script uses libcst for correct AST manipulation:
- Creates the file if it doesn't exist
- Appends to existing files with proper import merging
- Multiple page objects can share a module (e.g.,
ProductPageandCategoryPage)
Common base classes:
web_poet.WebPage— for pages using HTTP responses (most common)