mcpField
S
scraping

scraping

github.com/danielmiesslerData & databases19.6k107 installsUpdated 1d ago
Repository

About scraping

scraping is a skill for retrieving and extracting structured data from websites. It escalates from direct requests to browser rendering, proxies, and specialized APIs while emphasizing responsible request handling.

Key features

  1. 01Progressive retrieval methods
  2. 02Browser rendering for JavaScript sites
  3. 03Proxy rotation for restricted access
  4. 04Platform-specific scraper APIs
  5. 05Pagination and structured extraction

Use cases

  1. 01Collecting data from JavaScript-heavy pages
  2. 02Handling rate-limited or geo-restricted sites
  3. 03Extracting paginated website data
  4. 04Saving raw and processed results

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/danielmiessler/lifeos --skill scraping

After installing, type/scrapingin Claude Code to confirm it loaded.

Details

From the project README

Scraping Skill

Progressive Escalation

  1. Direct fetch — curl/fetch with standard headers
  2. Browser rendering — Headless browser for JS-heavy sites
  3. Proxy rotation — For rate-limited or geo-restricted content
  4. Specialized APIs — Platform-specific scrapers

Rules

  • Respect robots.txt
  • Rate limit requests
  • Handle pagination
  • Extract structured data
  • Store raw + processed versions

Comments

Markdown supported