mcpField
W
web-scraper

web-scraper

github.com/liranudiProductivity2250 installsUpdated 36w ago
Repository

About web-scraper

web-scraper is a set of scripts for searching, reading, browsing, and downloading web content. It suits users who need page extraction, persistent browser sessions, multiple tabs, and several output formats.

Key features

  1. 01Search web pages and return result snippets
  2. 02Extract pages in selectable formats
  3. 03Persistent browser sessions and multiple tabs
  4. 04Download files and extract PDF text

Use cases

  1. 01Find pages and collect their links and snippets
  2. 02Read pages as Markdown or plain text
  3. 03Inspect multiple pages through browser interactions
  4. 04Download PDFs and extract their text

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/liranudi/openclaw-web-scraper --skill web-scraper

After installing, type/web-scraperin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

.gitignore
README.md
scripts/
read_page.py
google_search.py
download_file.py
browser_session.py

Details

From the project README

Web Scraper

Four scripts, zero API keys. All output is JSON by default.

Dependencies: requests, beautifulsoup4, playwright (with Chromium). Optional: pdfplumber or PyPDF2 for PDF text extraction.

Install: pip install requests beautifulsoup4 playwright && playwright install chromium

1. Search the Web

python3 scripts/google_search.py "query" --pages N --engine ENGINE
  • --engine — duckduckgo (default), brave, or google
  • Returns [{title, url, snippet}, ...]

2. Read a Page (one-shot)

python3 scripts/read_page.py "https://url" [--max-chars N] [--visible] [--format json|markdown|text] [--no-dismiss]
  • --format — json (default), markdown, or text
  • Auto-dismisses cookie consent banners (skip with --no-dismiss)

3. Persistent Browser Session

python3 scripts/browser_session.py open "https://url"              # Open + extract
python3 scripts/browser_session.py navigate "https://other"        # Go to new URL
python3 scripts/browser_session.py extract [--format FMT]          # Re-read page
python3 scripts/browser_session.py screenshot [path] [--full]      # Save screenshot
python3 scripts/browser_session.py click "Submit"                  # Click by text/selector
python3 scripts/browser_session.py search "keyword"                # Search text in page
python3 scripts/browser_session.py tab new "https://url"           # Open new tab
python3 scripts/browser_session.py tab list                        # List all tabs
python3 scripts/browser_session.py tab switch 1                    # Switch to tab index
python3 scripts/browser_session.py tab close [index]               # Close tab
python3 scripts/browser_session.py dismiss-cookies                 # Manually dismiss cookies
python3 scripts/browser_session.py close                           # Close browser
  • Cookie consent auto-dismissed on open/navigate
  • Multiple tabs supported — open, switch, close independently
  • Search returns matching lines with line numbers
  • Extract supports json/markdown/text output

4. Download Files

python3 scripts/download_file.py "https://example.com/doc.pdf" [--output DIR] [--filename NAME]
  • Auto-detects filename from URL/headers
  • PDFs: extracts text if pdfplumber/PyPDF2 installed
  • Returns {status, path, filename, size_bytes, content_type, extracted_text}

Comments

Markdown supported