mcpField
P
playwright-scraper-skill

playwright-scraper-skill

github.com/csy12138daData & databases265 installsUpdated 33w ago
Repository

About playwright-scraper-skill

playwright-scraper-skill is a Playwright-based web scraping skill for workflows that range from static pages to JavaScript-driven and anti-bot-protected sites. It offers separate simple and stealth scripts, with configurable waits and browser output options.

Key features

  1. 01Methods matched to site protection
  2. 02Playwright scripts for dynamic pages
  3. 03Stealth measures for protected sites
  4. 04JSON, screenshot, and HTML output
  5. 05Environment-based script settings

Use cases

  1. 01Extracting content from JavaScript-driven pages
  2. 02Scraping pages with Cloudflare challenges
  3. 03Saving page screenshots and HTML
  4. 04Adjusting waits or browser visibility

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/csy12138da/playwright-scraper-skill --skill playwright-scraper-skill

After installing, type/playwright-scraper-skillin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

README.md
package.json
scripts/
playwright-simple.js
playwright-stealth.js

Details

From the project README

Playwright Scraper Skill

A Playwright-based web scraping skill with anti-bot protection. Choose the best approach based on the target website's anti-bot level.

🎯 Use Case Matrix

Target WebsiteAnti-Bot LevelRecommended MethodScript
Regular SitesLowweb_fetch toolN/A (built-in)
Dynamic SitesMediumPlaywright Simplescripts/playwright-simple.js
Cloudflare ProtectedHighPlaywright Stealth ⭐scripts/playwright-stealth.js

📦 Installation

cd playwright-scraper-skill
npm install
npx playwright install chromium

🚀 Quick Start

1️⃣ Simple Sites (No Anti-Bot)

Use built-in web_fetch tool for static sites.

2️⃣ Dynamic Sites (Requires JavaScript)

Use Playwright Simple:

node scripts/playwright-simple.js "https://example.com"
3️⃣ Anti-Bot Protected Sites (Cloudflare etc.)

Use Playwright Stealth:

node scripts/playwright-stealth.js "https://m.discuss.com.hk/#hot"

Features:

  • Hide automation markers (navigator.webdriver = false)
  • Realistic User-Agent (iPhone, Android)
  • Random delays to mimic human behavior
  • Screenshot and HTML saving support

📖 Script Descriptions

scripts/playwright-simple.js
  • Use Case: Regular dynamic websites
  • Speed: Fast (3-5 seconds)
  • Anti-Bot: None
  • Output: JSON (title, content, URL)
scripts/playwright-stealth.js ⭐
  • Use Case: Sites with Cloudflare or anti-bot protection
  • Speed: Medium (5-20 seconds)
  • Anti-Bot: Medium-High (hides automation, realistic UA)
  • Output: JSON + Screenshot + HTML file
  • Verified: 100% success on Discuss.com.hk

🔧 Customization

All scripts support environment variables:

# Set screenshot path
SCREENSHOT_PATH=/path/to/screenshot.png node scripts/playwright-stealth.js URL

# Set wait time (milliseconds)
WAIT_TIME=10000 node scripts/playwright-simple.js URL

# Enable headful mode (show browser)
HEADLESS=false node scripts/playwright-stealth.js URL

# Save HTML
SAVE_HTML=true node scripts/playwright-stealth.js URL

# Custom User-Agent
USER_AGENT="Mozilla/5.0 ..." node scripts/playwright-stealth.js URL

🛡️ Anti-Bot Techniques Summary

✅ Effective Anti-Bot Measures
  1. Hide navigator.webdriver — Essential
  2. Realistic User-Agent — Use real devices (iPhone, Android)
  3. Mimic Human Behavior — Random delays, scrolling
  4. Avoid Framework Signatures — Crawlee, Selenium are easily detected
  5. Use addInitScript (Playwright) — Inject before page load
❌ Ineffective Anti-Bot Measures
  1. Only changing User-Agent — Not enough
  2. Using high-level frameworks (Crawlee) — More easily detected
  3. Docker isolation — Doesn't help with Cloudflare

🔍 Troubleshooting

Issue: 403 Forbidden

Solution: Use playwright-stealth.js

Issue: Cloudflare Challenge Page

Solution:

  1. Increase wait time (10-15 seconds)
  2. Try headless: false (headful mode sometimes has higher success rate)
  3. Consider using proxy IPs

📚 References

Comments

Markdown supported