mcpField
S
scrape-add-page-object

scrape-add-page-object

github.com/zytedataDevelopment11169 installsUpdated 2d ago
Repository

About scrape-add-page-object

This skill adds an empty web-poet page object to a Scrapy project, creating a file or appending a class to an existing module. It supports automatic detection of required item fields, with an optional field list to override detection.

Key features

  1. 01Create or extend a Python module
  2. 02Add a page object with URL handling
  3. 03Detect required item fields automatically
  4. 04Override fields with a comma-separated list
  5. 05Merge imports when adding to existing files

Use cases

  1. 01Scaffold a page object for a new domain
  2. 02Add another page class to an existing module
  3. 03Generate stubs for required item fields
  4. 04Choose specific fields instead of automatic detection

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/zytedata/skills --skill scrape-add-page-object

After installing, type/scrape-add-page-objectin Claude Code to confirm it loaded.

Bundled filesscripts/add_page_object.py

Details

From the project README

You are adding an empty web-poet page object to a Scrapy project.

Read ${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md.

Input

The raw argument string is $ARGUMENTS. Split it into up to 6 whitespace-separated positional arguments:

  1. file_path: path to the .py file to create or append to (e.g. books_project/pages/books_toscrape_com.py)
  2. class_name: page object class name (e.g. ProductPage)
  3. domain: domain for @handle_urls (e.g. books.toscrape.com)
  4. base_class: base class import path (e.g. web_poet.WebPage)
  5. item_class: item class import path (e.g. books_project.items.ProductItem)
  6. fields: optional, comma-separated field names (e.g. name,price,rating)

Process

Run from the project root (the directory containing pyproject.toml) so the item class is importable for auto-detecting required fields:

uv run --project . --with libcst ${CLAUDE_SKILL_DIR}/scripts/add_page_object.py \
    FILE_PATH CLASS_NAME DOMAIN BASE_CLASS ITEM_CLASS

Required fields (those with no default in the item class) are detected automatically via itemadapter and get @field stubs. If all fields have defaults the class body is pass. Pass --fields name,price to override auto-detection.

The script uses libcst for correct AST manipulation:

  • Creates the file if it doesn't exist
  • Appends to existing files with proper import merging
  • Multiple page objects can share a module (e.g., ProductPage and CategoryPage)

Common base classes:

  • web_poet.WebPage — for pages using HTTP responses (most common)

Comments

Markdown supported