scraperapi-nodejs-sdk

ScraperAPI — Node.js SDK 最佳实践

要求： Node.js 14+、npm install scraperapi-sdk、SCRAPERAPI_API_KEY 环境变量。

设置

// CommonJS
const scraperapiClient = require('scraperapi-sdk')(process.env.SCRAPERAPI_API_KEY);

// ES Modules / TypeScript
import ScraperAPIClient from 'scraperapi-sdk';
const scraperapiClient = new ScraperAPIClient(process.env.SCRAPERAPI_API_KEY);

切勿将 API 密钥硬编码。每次都从环境变量中读取。

决策指南

情况	模式
单个 URL，立即需要结果	scraperapiClient.get(url)
20+ 个 URL 或批量任务	异步任务 API — 使用 fetch/axios 请求 async.scraperapi.com/jobs
支持的平台（Amazon、Google、Walmart、eBay、Redfin）	结构化数据端点 — api.scraperapi.com/structured/<vertical>
页面通过 JavaScript 加载内容	.get(url, { render: true })
网站屏蔽数据中心代理	.get(url, { premium: true })
需要向目标站点提交表单或 JSON 请求体	.post(url, options)

基本用法

// GET — returns HTML as a string (Promise-based)
const html = await scraperapiClient.get('https://example.com/');

// With parameters
const html = await scraperapiClient.get('https://example.com/', {
  render: true,
  country_code: 'us',
});

// Without async/await (Promise chain)
scraperapiClient.get('https://example.com/')
  .then(html => console.log(html))
  .catch(err => console.error(err));

POST 和 PUT 请求

// POST — forward a JSON body to the target site
const result = await scraperapiClient.post('https://example.com/api', {
  body: JSON.stringify({ query: 'example' }),
  headers: { 'Content-Type': 'application/json' },
});

// PUT — same signature, different HTTP method
const result = await scraperapiClient.put('https://example.com/resource', {
  body: JSON.stringify({ name: 'updated' }),
  headers: { 'Content-Type': 'application/json' },
});

参数参考

渲染

// Render JavaScript before returning HTML
// Use when: page uses React/Vue/Angular, or initial scrape returns empty content
// Cost: +10 credits
const html = await scraperapiClient.get('https://spa-site.com/', { render: true });

// Wait for a specific element before capturing (requires render: true)
const html = await scraperapiClient.get('https://spa-site.com/', {
  render: true,
  wait_for_selector: '.product-list',
});

// Screenshot (auto-enables rendering)
const html = await scraperapiClient.get('https://example.com/', { screenshot: true });

先不要设置 render: true。只有在响应缺少预期内容时才添加。

代理和地理定位

// Route through a specific country — no extra credit cost
const html = await scraperapiClient.get('https://example.com/', { country_code: 'gb' });

// Premium residential/mobile proxies
// Cost: 10 credits (25 with render: true)
const html = await scraperapiClient.get('https://hard-site.com/', { premium: true });

// Ultra-premium — for the toughest anti-bot protections
// Cost: 30 credits (75 with render: true)
// Incompatible with custom headers — keep_headers is ignored when ultra_premium is set
const html = await scraperapiClient.get('https://hardest-site.com/', { ultra_premium: true });

premium 和 ultra_premium 互斥，切勿同时设置。 升级顺序：standard（1 cr）→ render（10 cr）→ premium（10 cr）→ ultra_premium（30 cr）。

会话（粘性代理）

// Reuse the same proxy IP across requests (same session_number = same IP)
// Sessions expire 15 minutes after last use
const page1 = await scraperapiClient.get('https://example.com/page1', { session_number: 42 });
const page2 = await scraperapiClient.get('https://example.com/page2', { session_number: 42 });

标头和设备类型

// Forward your own headers to the target site
// Note: keep_headers is ignored when ultra_premium: true
const html = await scraperapiClient.get('https://example.com/', {
  keep_headers: true,
  headers: { 'Accept-Language': 'en-US', 'Referer': 'https://google.com' },
});

// Emulate mobile or desktop browser
const html = await scraperapiClient.get('https://example.com/', { device_type: 'mobile' });

响应格式和自动解析

// Structured JSON for supported sites (Amazon, Google, etc.)
const data = await scraperapiClient.get('https://amazon.com/dp/B09V3KXJPB', { autoparse: true });

// Markdown — useful for LLM pipelines or documentation tools
const md = await scraperapiClient.get('https://docs.example.com/', { output_format: 'markdown' });

// Plain text
const text = await scraperapiClient.get('https://example.com/', { output_format: 'text' });

账户信息

// Returns usage stats: concurrent request usage, total requests, account limits
const account = await scraperapiClient.account();
console.log(account);

可使用此功能通过程序监控积分消耗和并发限制。

升级阶梯

async function scrapeWithEscalation(url) {
  const tiers = [
    {},                                    // 1 credit
    { render: true },                      // 10 credits
    { premium: true },                     // 10 credits
    { premium: true, render: true },       // 25 credits
    { ultra_premium: true },               // 30 credits
  ];
  for (const params of tiers) {
    const html = await scraperapiClient.get(url, params);
    if (html && html.includes('<html')) return html;
  }
  return null;
}

异步任务（用于批量处理）

同步 SDK 会等待每次调用完成。对于 20+ 个 URL，请直接使用异步端点来并行提交任务。

const API_KEY = process.env.SCRAPERAPI_API_KEY;

async function submitJob(url, apiParams = {}) {
  const res = await fetch('https://async.scraperapi.com/jobs', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ apiKey: API_KEY, url, apiParams }),
  });
  return res.json(); // { id, status, statusUrl }
}

async function pollJob(job, maxWaitMs = 120_000, intervalMs = 5_000) {
  const deadline = Date.now() + maxWaitMs;
  while (Date.now() < deadline) {
    const res = await fetch(job.statusUrl);
    const data = await res.json();
    if (data.status === 'finished') return data.response.body;
    if (data.status === 'failed') throw new Error(`Job ${job.id} failed`);
    await new Promise(r => setTimeout(r, intervalMs));
  }
  throw new Error(`Job ${job.id} timed out`);
}

// Batch — submit all, then collect
const urls = ['https://example.com/p1', 'https://example.com/p2'];
const jobs = await Promise.all(urls.map(submitJob));
const results = await Promise.all(jobs.map(pollJob));

结构化数据端点

对于受支持的平台，请使用结构化端点，而非原始 HTML——它们会返回干净的 JSON， 无需编写解析逻辑。

async function structuredGet(vertical, params) {
  const query = new URLSearchParams({ api_key: API_KEY, ...params });
  const res = await fetch(`https://api.scraperapi.com/structured/${vertical}?${query}`);
  if (!res.ok) throw new Error(`${res.status} ${res.statusText}`);
  return res.json();
}

// Google SERP
const serp = await structuredGet('google/search', { query: 'javascript scraping' });

// Amazon product
const product = await structuredGet('amazon/product', { asin: 'B09V3KXJPB' });

// Walmart search
const items = await structuredGet('walmart/search', { query: 'standing desk' });

有关所有垂直领域和必填字段，请参阅结构化数据文档。

错误处理

async function safeScrape(url, params = {}) {
  try {
    return await scraperapiClient.get(url, params);
  } catch (err) {
    const status = err?.response?.status ?? err?.status;
    if (status === 401) throw new Error('Invalid API key — check SCRAPERAPI_API_KEY');
    if (status === 403) throw new Error('Blocked or out of credits — try premium or ultra_premium');
    if (status === 429) throw new Error('Rate limit — reduce concurrency or switch to async');
    if (status === 500 || status === 503) throw new Error('Transient error — retry with backoff');
    throw err;
  }
}

状态码：200 表示成功，401 表示密钥无效，403 表示被屏蔽或积分不足，404 表示未找到（目标）， 429 表示速率限制，500/503 表示暂时性错误（不收费，可安全重试）。

积分费用参考

请求类型	积分
标准	1
render: true	10
premium: true	10
premium: true + render: true	25
ultra_premium: true	30
ultra_premium: true + render: true	75

文档

• Node.js SDK 入门
• SDK 方法参考
• API 状态码
• 账户信息
• 控制面板和积分