mcpField
W
web-scraper

web-scraper

github.com/spacezephyrData & databases340281 installsUpdated 31w ago
Repository

About web-scraper

A Python script for fetching web pages and converting HTML into cleaned Markdown, intended for workflows that need page content. It supports timeout and output-length options, as well as raw HTML output.

Key features

  1. 01Fetch pages and convert to Markdown
  2. 02Set request timeouts
  3. 03Limit output length
  4. 04Option to return raw HTML
  5. 05Fetch multiple URLs in parallel

Use cases

  1. 01Extract articles for further processing
  2. 02Collect content from several pages
  3. 03Troubleshoot timeouts or empty results

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/spacezephyr/myskill --skill web-scraper

After installing, type/web-scraperin Claude Code to confirm it loaded.

Bundled filesscripts/fetch_url.py

Details

From the project README

Web Scraper

Fetch web page content and convert to clean markdown format.

Usage

Run the fetch script to get web content:

python3 scripts/fetch_url.py <url> [options]
Options
  • --timeout <seconds>: Request timeout (default: 30)
  • --max-length <chars>: Maximum output length (default: 100000)
  • --raw: Output raw HTML instead of markdown
Examples

Fetch single URL:

python3 scripts/fetch_url.py "https://example.com/article"

Fetch with custom timeout:

python3 scripts/fetch_url.py "https://example.com/article" --timeout 60

Fetch multiple URLs in parallel:

for url in "https://url1.com" "https://url2.com"; do
  python3 scripts/fetch_url.py "$url" &
done
wait

Workflow

  1. Single URL: Run fetch_url.py with the URL
  2. Multiple URLs: Run multiple fetch commands in parallel using background processes
  3. Handle errors: If a URL fails, check:
    • Network connectivity
    • URL validity
    • Website may block automated requests (try different User-Agent or use browser automation)

Output Format

The script converts HTML to clean markdown:

  • Headings → #, ##, ###, etc.
  • Lists → - for unordered, 1. for ordered
  • Bold/Italic → **bold**, *italic*
  • Code blocks preserved
  • Navigation, footer, and ads removed

Troubleshooting

403 Forbidden: Website blocks automated requests. Consider:

  • Some sites require JavaScript rendering (not supported by this script)
  • Try accessing from a different network

Timeout errors: Increase timeout with --timeout 60

Empty content: Website may require JavaScript to render content

Comments

Markdown supported