mcpField
S
scrape-ensure-project

scrape-ensure-project

github.com/zytedataDevelopment11167 installsUpdated 2d ago
Repository

About scrape-ensure-project

scrape-ensure-project is a skill for creating or checking a Scrapy project configured for web-poet page objects and Zyte API. It sets up a new project from a template or checks dependencies in an existing one.

Key features

  1. 01Create a project from a template
  2. 02Configure scrapy-poet and Zyte API
  3. 03Check dependencies in existing projects
  4. 04Ensure a fixtures directory exists

Use cases

  1. 01Starting a Scrapy project for page objects
  2. 02Checking an existing project's dependencies

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/zytedata/skills --skill scrape-ensure-project

After installing, type/scrape-ensure-projectin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

assets/project-template/
cookiecutter.json
assets/project-template/{{cookiecutter.project_name}}/
pyproject.toml
scrapy.cfg
assets/project-template/{{cookiecutter.project_name}}/{{cookiecutter.project_name}}/
__init__.py
items.py
settings.py
assets/project-template/{{cookiecutter.project_name}}/{{cookiecutter.project_name}}/pages/
__init__.py
assets/project-template/{{cookiecutter.project_name}}/{{cookiecutter.project_name}}/spiders/
__init__.py
assets/project-template/{{cookiecutter.project_name}}/fixtures/
.gitkeep

Details

From the project README

You are ensuring a Scrapy project exists and is properly configured for web-poet page objects and Zyte API.

Read ${CLAUDE_SKILL_DIR}/../scrape/references/python-environments.md.

Input

The raw argument string is $ARGUMENTS. Split it into 2 whitespace-separated positional arguments:

  1. project_dir: project directory (e.g. ./books-project)
  2. project_name: Python package name (e.g. books_project)

Process

1. Check if project already exists

If {project_dir}/pyproject.toml exists, this is an existing project. Go to step 3.

Otherwise, create the project from scratch (step 2).

2. Create new project

Use the bundled cookiecutter template:

uvx cookiecutter --no-input ${CLAUDE_SKILL_DIR}/assets/project-template -o PARENT_DIR project_name=PROJECT_NAME

This creates a Scrapy project with:

  • pyproject.toml with dependencies (scrapy, scrapy-poet, scrapy-zyte-api, web-poet)
  • scrapy.cfg with deploy settings
  • settings.py with scrapy-poet and Zyte API addons configured
  • pages/, spiders/, fixtures/ directories
  • items.py (empty, ready for item classes)

Then install:

cd PROJECT_DIR && uv sync

Report what was created and return.

3. Verify existing project dependencies

Read the project's pyproject.toml and check that these packages are in dependencies:

  • scrapy
  • scrapy-poet
  • web-poet
  • extruct
  • price-parser
  • pytest

If any are missing, warn the user — list the missing dependencies and ask whether to add them. Do NOT modify the project without confirmation.

Also check that fixtures/ directory exists (create it if missing — it's just a directory).

4. Report result

For new projects:

Created project at {project_dir}:
  - {project_name}/settings.py — scrapy-poet + Zyte API configured
  - {project_name}/items.py — ready for item classes
  - {project_name}/pages/ — ready for page objects
  - fixtures/ — ready for test fixtures

For existing projects:

Using existing project at {project_dir}.

Or if dependencies are missing:

Using existing project at {project_dir}.
Warning: missing dependencies: {list}
Add them to pyproject.toml?

Comments

Markdown supported