scrape-spec

You are expanding and validating an extraction spec that was drafted by /scrape-define. Download diverse detail and listing pages, compare HTML variants, extract values, and optionally present for browser-based review.

Read ${CLAUDESKILLDIR}/../scrape/references/python-environments.md.

The input is a spec folder with at least one data type containing a schema and 1 page (from /scrape-define). The output is the same folder with more pages, validated values, and a navigation data type.

Input

The raw argument string is $ARGUMENTS — a single value, used as-is:

1. site_path: path to the site spec folder, e.g. .scrape/books-toscrape

Step 1: Read existing spec

Read {sitepath}/spec.json to get url and datatypes.

For the primary data type (first in data_types that isn't "navigation"), read:

• {sitepath}/{datatype}/spec.json → schema, html_variant

Derive sitename from sitepath (last component, e.g. books-toscrape).

Step 2: Explore the site

Stage 1 doesn't store pages — only schema and values. Stage 2 downloads a small fresh sample using /scrape-explore-site in a subagent.

Agent(description="explore-site", prompt="Run /scrape-explore-site {url} .scrape/.work/{site_name}/explore 2 2")

This downloads the start page + up to 2 detail pages + up to 2 list pages, classifies links, and generates navigation values. All output goes to .scrape/.work/{site_name}/explore/.

If the subagent reports that the site is blocked, invoke /scrape-zyte-login. After it returns, re-run the scrape-explore-site subagent above. Only proceed once exploration succeeds.

Then distribute pages to the right data-type subfolders:

mkdir -p {site_path}/{data_type}/pages {site_path}/navigation/pages {site_path}/navigation/values

# Detail pages → data type
for d in .scrape/.work/{site_name}/explore/pages/detail-*; do
  [ -d "$d" ] && cp -r "$d" {site_path}/{data_type}/pages/
done

# Start + list pages → navigation
for d in .scrape/.work/{site_name}/explore/pages/start-* .scrape/.work/{site_name}/explore/pages/list-*; do
  [ -d "$d" ] && cp -r "$d" {site_path}/navigation/pages/
done

# Navigation values (generated by explore-site)
cp .scrape/.work/{site_name}/explore/values/*.json {site_path}/navigation/values/ 2>/dev/null || true

Step 3: Determine navigation HTML variant

Before analyzing detail pages, independently determine the HTML variant navigation pages need. Run extract_links.py on all navigation pages using both variants:

mkdir -p .scrape/.work/{site_name}/analyze-nav

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_links.py \
  {site_path}/navigation/pages/*/raw.html \
  --group --base-url-from-meta \
  > .scrape/.work/{site_name}/analyze-nav/nav.raw.json 2>/dev/null || true

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_links.py \
  {site_path}/navigation/pages/*/rendered.html \
  --group --base-url-from-meta \
  > .scrape/.work/{site_name}/analyze-nav/nav.rendered.json 2>/dev/null || true

Read both output files and analyze the link groups. Determine which HTML variant provides better navigation coverage — consider the group structure, whether meaningful navigation links appear in each variant, and whether one variant reveals links the other misses. Prefer raw unless there is clear evidence that rendered provides materially better navigation coverage.

Store the result as navhtmlvariant for use in Step 6.

Step 4: Analyze detail sample and list pages in parallel

First analyze a bounded sample of detail pages with both HTML variants. Use the first detail page that has both raw.html and rendered.html when list pages are present; otherwise use the first 1-2 detail pages. This means the detail-page variant comparison uses at most 1 page when list pages are also being analyzed, and at most 2 pages when no list pages are available. Do not launch more than 4 detail-page analysis agents for the variant comparison. This keeps the workflow responsive while still testing whether raw or rendered can extract the approved schema fields.

Also analyze all list pages from {site_path}/navigation/pages/ with the raw variant using the data type schema. Launch the detail sample agents plus one Agent per list page in a single message for parallel execution. Keep the combined launch at 5 Agents or fewer to avoid provider thread limits; if there are list pages, prefer detail-1 raw/rendered plus the list-page agents over analyzing a second detail page. Issue all the Agent calls in one response with no tool calls in between — do not wait for any result before launching the rest.

Before launching the agents, create the output directories:

mkdir -p .scrape/.work/{site_name}/analyze-page
mkdir -p {site_path}/{data_type}-list/values

Agent(description="analyze detail-1 raw", prompt="/scrape-analyze-page Extract data from {site_path}/{data_type}/pages/detail-1/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-1.raw.json ")
Agent(description="analyze detail-1 rendered", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-1/rendered.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-1.rendered.json")
Agent(description="analyze list-1", prompt="/scrape-analyze-page --list-mode Extract all {data_type} items from {site_path}/navigation/pages/list-1/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into {site_path}/{data_type}-list/values/list-1.json")
Agent(description="analyze list-2", prompt="/scrape-analyze-page --list-mode Extract all {data_type} items from {site_path}/navigation/pages/list-2/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into {site_path}/{data_type}-list/values/list-2.json")
... (all list pages in the same message)

If there are no list pages, add the second detail-page pair instead:

Agent(description="analyze detail-2 raw", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-2/raw.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-2.raw.json")
Agent(description="analyze detail-2 rendered", prompt="Run /scrape-analyze-page {site_path}/{data_type}/pages/detail-2/rendered.html using the schema in {site_path}/{data_type}/spec.json and save it into .scrape/.work/{site_name}/analyze-page/detail-2.rendered.json")

Skip variants whose HTML files don't exist. If there are no list pages in {sitepath}/navigation/pages/, skip the list-page agents and the mkdir for {datatype}-list/values. The schema_path gives analyze-page the approved field names, descriptions, and examples — so it extracts with the correct names and value formats. After Step 5 chooses the variant, do not launch extra detail-page analyses. Do not wait for every list-page agent before continuing: once the bounded detail sample is complete and at least one list-page values file is available, proceed to Step 4.5 using the completed list outputs so slow optional list analyses do not block finalization.

Step 4.5: Decide whether to create a list-page data type

This step runs only if list pages were found and analyzed in Step 4.

Read the completed {sitepath}/{datatype}-list/values/list-*.json files. For each completed file, check whether every source: "requested" field from the schema appears as a key in at least one item in the values array. If no list values file has completed yet, set listspeccreated = false and proceed with detail-page extraction rather than waiting.

If ALL requested fields are found in EVERY completed list-page values file:

Set listspeccreated = true.

1. Write {sitepath}/{datatype}-list/spec.json:

``json { "url": "{url}", "datatype": "{datatype}-list", "htmlvariant": "raw", "schema": {<same schema object as in {sitepath}/{data_type}/spec.json>} } ``

1. Copy list pages into the new data type:

``bash mkdir -p {sitepath}/{datatype}-list/pages for d in {sitepath}/navigation/pages/list-*; do [ -d "$d" ] && cp -r "$d" {sitepath}/{data_type}-list/pages/ done ``

Otherwise: set listspeccreated = false. Proceed with detail-page extraction as normal — the {data_type}-list/values/ directory may remain but will be ignored.

Step 5: Choose HTML variant

Compare raw vs rendered results across the bounded sample from .scrape/.work/{site_name}/analyze-page/.

For each page, compare {pageid}.raw.json and {pageid}.rendered.json:

• Which variant found more of the schema fields?
• Which fields are only in one variant?
• Do values differ between variants for the same field?

Use a single small Python script for this comparison. The script should load {sitepath}/{datatype}/spec.json, enumerate .scrape/.work/{site_name}/analyze-page/*.json, compare only schema field coverage and values, print a concise summary, and recommend raw unless rendered consistently finds requested schema fields that raw misses.

Raw is preferred by default — it's faster, cheaper, and more reliable. Only use rendered if it finds schema fields that raw consistently misses, or if raw HTML is essentially empty (SPA site).

Present the comparison in the terminal:

HTML variant comparison:
  Both variants found: name, price, brand, description, image_url
  Only in rendered: reviews_count (3/3 pages), sale_price (2/3 pages)
  Only in raw: (none)
  Value differences:
    price: raw="$29.99", rendered="$24.99" (detail-1) — rendered may show sale price

  Recommendation: raw (all schema fields found)
  Note: rendered also has reviews_count, sale_price — say "use rendered" if you need these.

If it's clear which variant to use, use it. If it's not, for example when the raw view contains most of the fields but not all of them, ask the user via AskUserQuestion which variant to use, providing details:

• question: "Which HTML variant should I use for extraction?"
• header: "HTML variant"
• options: "Use raw" and "Use rendered", mark the recommended one as such.

In this case use the variant that the user selects.

If both variants find all approved schema fields in the bounded sample, choose raw immediately. If the variant changes from what Stage 1 used, update {sitepath}/{datatype}/spec.json with the new html_variant. Once the variant is chosen, continue directly to Step 6; do not start more analysis agents.

Step 6: Extract values (skip if listspeccreated)

If listspeccreated is true, skip this step — values were already written to {data_type}-list/values/ in Step 4.

Otherwise, use extract_values.py to build values from analysis files, filtered by the schema:

uv run ${CLAUDE_SKILL_DIR}/../scrape-explore-site/scripts/extract_values.py \
  .scrape/.work/{site_name}/analyze-page/ \
  {site_path}/{data_type}/spec.json \
  --variant {html_variant} \
  -O {site_path}/{data_type}/values/

This overwrites any existing values files (including the one from Stage 1) with fresh extractions from the analyzed detail-page sample. No --renames needed — Stage 2's analyze-page receives the full schema (names + descriptions + examples), so it extracts with the correct field names directly.

Navigation

Write {sitepath}/navigation/spec.json with the fixed navigation schema from ${CLAUDESKILLDIR}/../scrape/references/extraction-spec.md, using the site URL and navhtml_variant (determined in Step 3).

Navigation values were already copied from explore-site output in Step 2.

Step 7: Optional browser review

Tell the user the extraction stats first ("Extracted values for {N} detail pages and {M} navigation pages."), then ask via AskUserQuestion whether they want a browser review.

If the environment cannot ask follow-up questions, skip browser review and continue.

• question: "Open a browser review of the extracted values?"
• header: "Browser review"
• options:

- Skip browser review — "Continue without opening the browser." - Open browser review — "Review the values in the browser."

If the user picks Skip review, go to step 9.

If the user picks Open browser review, invoke /scrape-review-schema with the data type spec:

/scrape-review-schema {site_path}/{data_type} .scrape/.work/{site_name} {schema_json} {html_variant}

Report the review dir path to the user before opening.

Step 8: Apply feedback

If the user reviews and provides feedback:

If feedback starts with APPROVED: apply any included schema changes (drops, renames, description edits, kept fields → change source to "requested") and skip to step 9. The user has signed off.

If feedback does NOT start with APPROVED (user clicked "Request changes"):

• Apply schema changes (drops, renames, description edits) directly
• For value corrections: check the other variant's analysis files first. If the other variant has the correct value, suggest switching variants:

`` You corrected price to "$24.99" — the rendered variant already extracts this correctly (across 3/3 pages). Switch to rendered? [Y/n] `` If the user agrees, switch variant, update spec, and regenerate values.

• If the other variant doesn't help: re-run analysis for corrected fields across ALL detail pages (using the chosen variant). Launch parallel Agents, one per page.

After re-analysis or variant switch, re-extract values (step 6) and offer review again. Pass a changes summary as the 5th argument to /scrape-review-schema:

/scrape-review-schema {site_path}/{data_type} .scrape/.work/{site_name} {schema_json} {html_variant} '["Re-analyzed price across all pages","Dropped field isbn"]'

Loop steps 7-8 until the user approves or skips review.

Step 9: Finalize

Update {sitepath}/{datatype}/spec.json with any schema changes from the review (only relevant when listspeccreated is false, since detail pages were used).

Update {sitepath}/spec.json — set datatypes as follows:

• If listspeccreated is true: ["{datatype}-list", "navigation"] — omit "{datatype}" since detail-page extraction is not needed.
• Otherwise: ["{data_type}", "navigation"]

Report:

If listspeccreated is true:

Spec finalized at {site_path}/:
  {data_type}-list: {N} list pages, {F} fields, {K} items extracted
  navigation: {M} pages

All requested fields found on list pages — skipping detail-page extraction.
Ready for codegen: /scrape-codegen {site_path}/{data_type}-list ./{project_dir}

Otherwise:

Spec finalized at {site_path}/:
  {data_type}: {N} detail pages, {F} fields
  navigation: {M} pages

Ready for codegen: /scrape-codegen {site_path}/{data_type} ./{project_dir}