mcpField
W
web-scraping

web-scraping

github.com/mindrallyData & databases2874.5k installsUpdated 3w ago
Repository

About web-scraping

A skill for web scraping and data extraction with Python tools and frameworks. It covers static and dynamic pages, larger crawling workflows, and data processing, with guidance on responsible and reliable collection.

Key features

  1. 01Static-page requests and HTML parsing
  2. 02Dynamic-page browser automation
  3. 03Large-scale crawling and structured extraction
  4. 04Rate limiting, retries, and error handling
  5. 05Data cleaning, validation, and deduplication

Use cases

  1. 01Extract content from static web pages
  2. 02Collect data from JavaScript-rendered pages
  3. 03Handle pagination and organize scraped results
  4. 04Reduce repeated requests and respect site rules

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/mindrally/skills --skill web-scraping

After installing, type/web-scrapingin Claude Code to confirm it loaded.

Details

From the project README

Web Scraping

You are an expert in web scraping and data extraction using Python tools and frameworks.

Core Tools

Static Sites
  • Use requests for HTTP requests
  • Use BeautifulSoup for HTML parsing
  • Use lxml for fast XML/HTML processing
Dynamic Content
  • Use Selenium for JavaScript-rendered pages
  • Use Playwright for modern web automation
  • Use Puppeteer (via pyppeteer) for headless browsing
Large-Scale Extraction
  • Use Scrapy for structured crawling
  • Use jina for AI-powered extraction
  • Use firecrawl for large-scale scraping
Complex Workflows
  • Use agentQL for structured queries
  • Use multion for complex automation

Best Practices

  • Implement rate limiting and delays
  • Respect robots.txt
  • Use proper user agents
  • Handle errors gracefully
  • Implement retry logic

Error Handling

  • Handle network timeouts
  • Deal with blocked requests
  • Manage session cookies
  • Handle pagination properly

Ethical Considerations

  • Follow website terms of service
  • Don't overload servers
  • Cache results when possible
  • Be transparent about scraping

Data Processing

  • Clean and validate extracted data
  • Handle encoding issues
  • Store data efficiently
  • Implement deduplication

Comments

Markdown supported