webextrator

WebExtrator Web Render & Extract

Render and extract web content through AceDataCloud's WebExtrator API — real headless Chromium plus a three-tier extraction pipeline (schema.org JSON-LD mapper → LLM typed extractor → Readability/markdown fallback).

Setup: See authentication for token setup.

Quick Start

curl -X POST https://api.acedata.cloud/webextrator/extract \
  -H "Authorization: Bearer $ACEDATACLOUD_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "expected_type": "general"}'

Returns synchronously in seconds — no task polling needed.

Endpoints

Path	Purpose
POST /webextrator/render	Headless Chromium render → raw HTML + clean text + title
POST /webextrator/extract	Render + structured extraction (schema.org + LLM types) + markdown
POST /webextrator/tasks	Look up historical render/extract task envelopes (7-day retention, free)

Workflows

1. Extract typed content

POST /webextrator/extract
{
  "url": "https://example.com",
  "expected_type": "general"
}

Real response (trimmed):

{
  "success": true,
  "task_id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5",
  "trace_id": "f2a7c0b0-c17c-4bc9-b6e7-9c59746dd366",
  "elapsed": 0.003,
  "data": {
    "kind": "extract",
    "url": "https://example.com",
    "finalUrl": "https://example.com/",
    "contentType": "general",
    "title": "Example Domain",
    "description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.",
    "language": "en",
    "images": [],
    "links": [],
    "markdown": "...",
    "text": "...",
    "structured": { "schemaOrg": {}, "openGraph": {}, "jsonLd": [] }
  }
}

2. Render raw HTML

POST /webextrator/render
{
  "url": "https://example.com",
  "wait_until": "networkidle",
  "block_resources": ["image", "media", "font"]
}

Returns data.html, data.text, data.title, data.status, data.finalUrl.

3. Look up a task

POST /webextrator/tasks
{
  "action": "retrieve",
  "id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5"
}

Parameters

Render & Extract (shared)

Parameter	Required	Description
url	Yes	Page URL to render (http(s)://)
wait_until	No	load / domcontentloaded / networkidle / commit (default networkidle)
timeout	No	Navigation timeout in seconds (default 30)
delay	No	Extra wait in seconds after wait_until (for SPAs)
wait_for_selector	No	CSS selector to wait for before ready
block_resources	No	Drop image/font/media/stylesheet/xhr/fetch
headers	No	Extra request headers for the target site
user_agent	No	Override the browser User-Agent
async	No	true submits without blocking; poll /webextrator/tasks for the result
callback_url	No	Posted the final envelope when running asynchronously

Extract-only

Parameter	Required	Description
expected_type	No	product / article / general — skips the heuristic
enable_llm	No	Allow LLM extractor when schema.org found nothing (default false)

Tasks

Parameter	Required	Description
action	Yes	retrieve (single) or retrieve_batch (many)
id / trace_id	one of	For retrieve
ids / trace_ids	one of	For retrieve_batch

Gotchas

• Parameters use snake_case (wait_until, block_resources), not camelCase
• Cache hits are still billed; identical URLs return in ~0.003s
• expected_type only allows product/article/general — typed kinds (recipe/video/job) are detected automatically from schema.org
• enable_llm has no effect when the page ships schema.org JSON-LD — the deterministic mapper wins for free
• Tasks API is free and retains records for 7 days only

MCP: pip install mcp-webextrator | Hosted: https://webextrator.mcp.acedata.cloud/mcp | See all MCP servers