webpage-screenshotter

Screenshotter Skill

描述

基于 Playwright 的高分辨率截图捕获技能，可使用优化后的设置，为任意 URL 截取整页截图，兼顾画质与可靠性。

功能

• 高分辨率视口（1920x1080）
• 整页截图捕获
• 超时错误处理
• 重新加载页面以提高稳定性
• 将截图数据编码为 Base64
• 为加载缓慢的页面延长超时时间（120 秒）

配置

• 视口：1920x1080 像素
• 设备缩放系数：0.5
• 超时时间：120 秒
• 等待策略：domcontentloaded
• 截图类型：整页

等待策略

根据需要选择合适的等待策略：

• domcontentloaded（默认）：速度快，等待 HTML 解析完成。适用于大多数页面。
• load：等待所有资源（图片、样式表）加载完成。更可靠，但速度较慢。
• networkidle：等待网络活动持续 500ms 无变化。最适合动态内容。

Python 实现

import asyncio
import base64
from playwright.async_api import async_playwright
import playwright._impl._api_types

async def get_screenshot(url):
    """
    Capture a full-page screenshot of a given URL using Playwright.
    
    Args:
        url (str): The URL to capture
        
    Returns:
        str: Base64-encoded screenshot data
    """
    print('in get_screenshot_func_remote', url)

    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(viewport={"width": 1920, "height": 1080, "device_scale_factor": 0.5})
        
        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=120000)
        except playwright._impl._api_types.TimeoutError:
            print(f"TimeoutError: Failed to load {url} within the specified timeout.")
            await asyncio.sleep(2)
        
        # Reload page for stability
        await page.reload(wait_until='domcontentloaded')
        
        # Capture full-page screenshot
        await page.screenshot(path="screenshot.png", full_page=True)
        await browser.close()
        
        # Read and encode screenshot
        data = open("screenshot.png", "rb").read()
        print('screenshot done,', len(data))
        encoded_data = base64.b64encode(data).decode('utf-8')
        base64_image_data = f"data:image/png;base64,{encoded_data}"
        print("Screenshot of size %d bytes" % len(data))
        
        return encoded_data

使用示例

import asyncio

# Basic usage
async def main():
    url = "https://example.com"
    screenshot_data = await get_screenshot(url)
    print(f"Screenshot captured and encoded: {len(screenshot_data)} characters")

# Run the async function
asyncio.run(main())

高级用法

保存到自定义路径

async def get_screenshot_custom_path(url, output_path="screenshot.png"):
    """
    Capture screenshot with custom output path.
    """
    print('in get_screenshot_func_remote', url)

    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(viewport={"width": 1920, "height": 1080, "device_scale_factor": 0.5})
        
        try:
            await page.goto(url, wait_until="domcontentloaded", timeout=120000)
        except playwright._impl._api_types.TimeoutError:
            print(f"TimeoutError: Failed to load {url} within the specified timeout.")
            await asyncio.sleep(2)
        
        await page.reload(wait_until='domcontentloaded')
        await page.screenshot(path=output_path, full_page=True)
        await browser.close()
        
        data = open(output_path, "rb").read()
        print('screenshot done,', len(data))
        encoded_data = base64.b64encode(data).decode('utf-8')
        print("Screenshot of size %d bytes" % len(data))
        
        return encoded_data

批量截图

async def capture_multiple_screenshots(urls):
    """
    Capture screenshots of multiple URLs.
    
    Args:
        urls (list): List of URLs to capture
        
    Returns:
        dict: Dictionary mapping URLs to their base64-encoded screenshots
    """
    results = {}
    
    for url in urls:
        try:
            screenshot_data = await get_screenshot(url)
            results[url] = screenshot_data
        except Exception as e:
            print(f"Error capturing {url}: {e}")
            results[url] = None
    
    return results

# Usage
urls = ["https://example.com", "https://another-site.com"]
results = asyncio.run(capture_multiple_screenshots(urls))

等待整页加载完成

async def get_screenshot_full_load(url):
    """Wait for all resources to load before screenshot."""
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(viewport={"width": 1920, "height": 1080, "device_scale_factor": 0.5})
        
        # Wait for complete load including all resources
        await page.goto(url, wait_until="load", timeout=120000)
        
        await page.screenshot(path="screenshot.png", full_page=True)
        await browser.close()
        
        data = open("screenshot.png", "rb").read()
        return base64.b64encode(data).decode('utf-8')

等待网络空闲（动态内容）

async def get_screenshot_network_idle(url):
    """Wait for network to be idle - best for JavaScript-heavy sites."""
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(viewport={"width": 1920, "height": 1080, "device_scale_factor": 0.5})
        
        # Wait for network idle (no requests for 500ms)
        await page.goto(url, wait_until="networkidle", timeout=120000)
        
        # Optional: wait for specific element
        await page.wait_for_selector("body", state="visible")
        
        await page.screenshot(path="screenshot.png", full_page=True)
        await browser.close()
        
        data = open("screenshot.png", "rb").read()
        return base64.b64encode(data).decode('utf-8')

绕过 Cloudflare

对于受 Cloudflare 保护的网站，标准 Playwright 会话通常会被检测到。可使用以下技巧绕过检测：

安装

Node.js（JavaScript）：

npm install playwright-extra playwright-extra-plugin-stealth

Python：

pip install playwright playwright-stealth

隐身模式设置（JavaScript）

const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();

// CRITICAL: Must use stealth plugin BEFORE launching browser
chromium.use(stealth);

// Launch with stealth enabled
const browser = await chromium.launch({ 
  headless: false // Headed mode reduces detection
});

浏览器指纹随机化

随机更改视口、user-agent、区域设置和时区，以避免被识别：

const context = await browser.newContext({
  viewport: {
    width: 1280 + Math.floor(Math.random() * 100), // Randomize
    height: 720 + Math.floor(Math.random() * 100)
  },
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
  locale: 'en-US',
  timezoneId: 'America/New_York'
});

持久化会话

重用 cookies 和 localStorage，让访问看起来像是回访用户：

const userDataDir = './session-profile';

const browser = await chromium.launchPersistentContext(userDataDir, {
  headless: false,
  args: ['--start-maximized']
});

代理轮换

轮换代理以分散请求，避免基于 IP 的拦截：

const browser = await chromium.launch({
  headless: false,
  args: [
    '--proxy-server=http://username:password@proxy-ip:port'
  ]
});

CAPTCHA 检测

// Check for CAPTCHA iframe
const isCaptchaPresent = await page.$('iframe[src*="captcha"]');

if (isCaptchaPresent) {
  console.log('CAPTCHA detected – solve or switch proxy');
}

CAPTCHA 求解（可选）

对于 reCAPTCHA，可使用 2Captcha 服务：

const RecaptchaPlugin = require('@extra/recaptcha');

chromium.use(
  RecaptchaPlugin({
    provider: {
      id: '2captcha',
      token: 'YOUR_2CAPTCHA_API_KEY'
    },
    visualFeedback: true
  })
);

await page.solveRecaptchas();

会话 Cookie 管理

保存并恢复 cookies，以保持会话连续性：

// Save cookies after successful scrape
const cookies = await context.cookies();
fs.writeFileSync('./cookies.json', JSON.stringify(cookies, null, 2));

// Restore cookies on next run
const savedCookies = JSON.parse(fs.readFileSync('./cookies.json'));
await context.addCookies(savedCookies);

完整的 Cloudflare 绕过示例

const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();
const fs = require('fs');

chromium.use(stealth);

async function screenshotWithCloudflareBypass(url, proxy = null) {
  const args = proxy ? [`--proxy-server=${proxy}`] : [];
  
  const browser = await chromium.launch({
    headless: false,
    args: args
  });

  const context = await browser.newContext({
    viewport: {
      width: 1280 + Math.floor(Math.random() * 100),
      height: 720 + Math.floor(Math.random() * 100)
    },
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    locale: 'en-US',
    timezoneId: 'America/New_York'
  });

  const page = await context.newPage();

  // Load page and wait for Cloudflare checks
  await page.goto(url, { waitUntil: "domcontentloaded" });
  await page.waitForTimeout(5000); // Let Cloudflare finish background checks

  // Check for CAPTCHA
  const captchaPresent = await page.$('iframe[src*="captcha"]');
  if (captchaPresent) {
    console.log('CAPTCHA detected');
    // Handle CAPTCHA or switch proxy
  }

  // Capture screenshot
  await page.screenshot({ path: "screenshot.png", fullPage: true });

  // Save cookies for next visit
  const cookies = await context.cookies();
  fs.writeFileSync('./cookies.json', JSON.stringify(cookies, null, 2));

  await browser.close();
}

最佳实践

1. 使用有头模式（headless: false）——降低被检测的概率
2. 轮换代理——避免基于 IP 的拦截
3. 随机化指纹——视口、user-agent、时区
4. 持久化会话——重用 cookies，让访问看起来像是回访用户
5. 等待 Cloudflare——添加延迟，以便完成后台 JS 检查
6. 监控 CAPTCHA——检测并处理验证挑战
7. 限制重复使用——不要过于频繁地重用同一组代理和 UA

依赖项

Python：

pip install playwright
playwright install chromium

Node.js（含 Cloudflare 绕过功能）：

npm install playwright-extra playwright-extra-plugin-stealth

错误处理

该技能包含完善的错误处理机制，可应对：

• 超时错误：妥善处理未能在 120 秒内加载的页面
• 网络故障：即使初始页面加载失败，也会继续执行
• 浏览器崩溃：即使发生错误，也能确保正确关闭浏览器

性能说明

• 视口设置为 1920x1080，设备缩放系数为 0.5，实际渲染分辨率为 960x540
• 对于特别长的页面，整页截图可能需要更长时间
• 重新加载页面可确保动态内容完全加载
• 截图会先临时保存为 PNG 文件，然后编码为 Base64

使用场景

• 自动化网站监控
• 视觉回归测试
• 带视觉确认的网页抓取
• 文档生成
• 网页归档
• 质量保证工作流