mcpField
W
web-scraper

web-scraper

github.com/guia-matthieuData & databases150583 installsUpdated 4w ago
Repository

About web-scraper

A command-line web scraping skill for collecting structured information from websites, intended for research and data collection tasks. It uses BeautifulSoup and requests to extract selected elements, links, emails, and schema-based content.

Key features

  1. 01Extract elements with CSS selectors
  2. 02Collect internal or external links
  3. 03Find email addresses
  4. 04Extract article or product data
  5. 05Scraping guidance for responsible use

Use cases

  1. 01Comparing competitor pricing
  2. 02Collecting directory contact details
  3. 03Auditing page headings and metadata
  4. 04Gathering article content from websites

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/guia-matthieu/clawfu-skills --skill web-scraper

After installing, type/web-scraperin Claude Code to confirm it loaded.

Bundled filesscripts/main.pyscripts/requirements.txt

Details

From the project README

Web Scraper

Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.

When to Use This Skill

  • Competitor research - Scrape pricing, features, positioning
  • Lead generation - Extract contact info from directories
  • Content audit - Pull headings, links, meta data
  • Price monitoring - Track competitor pricing changes
  • Data collection - Gather research data from multiple sources

What Claude Does vs What You Decide

Claude DoesYou Decide
Structures analysis frameworksStrategic priorities
Synthesizes market dataCompetitive positioning
Identifies opportunitiesResource allocation
Creates strategic optionsFinal strategy selection
Suggests implementation approachesExecution decisions

Dependencies

pip install beautifulsoup4 requests pandas click lxml

Commands

Scrape Elements
python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"
Extract Links
python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only
Extract Emails
python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2
Extract Structured Data
python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product

Examples

Example 1: Scrape Competitor Pricing
python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"

# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us
Example 2: Extract Article Content
python scripts/main.py structured https://blog.example.com/post --schema article

# Output: article_data.json
# {
#   "title": "How to Scale Your Startup",
#   "author": "Jane Doe",
#   "date": "2024-01-15",
#   "content": "...",
#   "word_count": 1523
# }

CSS Selector Reference

SelectorDescriptionExample
tagElement typeh1, p, div
.classClass name.price, .title
#idElement ID#main-content
tag.classTag with classdiv.product
tag[attr]Has attributea[href]
parent > childDirect childul > li
tag1, tag2Multipleh1, h2, h3

Ethical Scraping Guidelines

  1. Check robots.txt - Respect site's scraping policy
  2. Rate limit - Don't overload servers (1-2 req/sec)
  3. Identify yourself - Use descriptive User-Agent
  4. Cache requests - Don't re-scrape unchanged pages
  5. Terms of Service - Check if scraping is allowed

Skill Boundaries

What This Skill Does Well
  • Structuring strategic analysis
  • Identifying market opportunities
  • Creating strategic frameworks
  • Synthesizing competitive data
What This Skill Cannot Do
  • Replace market research
  • Guarantee strategic success
  • Know proprietary competitor info
  • Make executive decisions

Related Skills

Skill Metadata

  • Mode: centaur
category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week

Comments

Markdown supported