webclaw

webclaw

Web extraction for AI agents, powered by a local Rust engine. It turns any URL into clean Markdown, text, or JSON through the webclaw MCP server, and it works with no API key.

Install

npx create-webclaw

This writes the npx @webclaw/mcp MCP config into your agent (Claude Code, Cursor, Windsurf, Codex, Antigravity, and more) — nothing to install; the server is fetched and cached on first launch. Restart the agent to load the tools below.

• No key needed. Extraction runs on your machine: free, private, and unlimited for the common case. Static sites, docs, blogs, server-rendered pages, and product pages all work locally.
• Optional upgrade. Set WEBCLAW_API_KEY (from https://webclaw.io) and webclaw escalates to the hosted engine for the pages local extraction can't finish: bot-protected sites and JavaScript-rendered SPAs. Without a key, those pages return a clear message that tells you how to unlock them.

When to use this skill

• You need reliable web content with clean, structured output.
• web_fetch returns empty, truncated, or blocked content.
• You need structured data (pricing tables, product specs, contact info) as JSON.
• You need to crawl a whole site or discover every URL.
• You need LLM-optimized content, cleaner than raw markdown.
• You need a site-specific extractor (GitHub, Reddit, YouTube, npm, PyPI, Amazon).
• You need to summarize, diff, or search the web.

Tools

All tools run locally with no key unless noted. Output formats: markdown (default), text, llm (adds a title and URL header with clean link references, best for feeding to a model), and json (full metadata).

scrape: extract a single URL

url (required), format, include_selectors, exclude_selectors, only_main_content, browser (chrome | firefox | random), cookies. YouTube watch, shorts, and youtu.be URLs also return a transcript and a video-metadata block alongside the content.

crawl: scrape an entire site

url, depth (default 2), max_pages (default 50), concurrency (default 5), use_sitemap, format.

map: discover URLs

url. Sitemap-first discovery, with a bounded same-origin crawl fallback when the sitemap is thin.

batch: many URLs in parallel

urls (array), format, concurrency (default 5).

extract: structured data via LLM

url, plus either prompt (natural language) or schema (a JSON schema). See LLM setup below.

summarize: quick summary

url, max_sentences (default 3). See LLM setup below.

diff: detect content changes

url, previous_snapshot (a prior extraction as JSON). Compares at the extracted-content level rather than raw HTML.

brand: visual identity

url. Returns colors, fonts, logo, and favicon.

search: web search

query, num_results (≤10), country, lang, scrape (also fetch and extract each result page). Uses your own SERPER_API_KEY (free at serper.dev) locally, and falls back to the hosted API when unset.

vertical_scrape: typed JSON for a specific site

name (extractor name), url. Returns typed fields (title, price, author, rating) instead of generic markdown, or a clear "URL mismatch" error when the URL doesn't fit the extractor. Verticals for protected sites (Amazon, eBay, Etsy, Trustpilot) need WEBCLAW_API_KEY; without one they return a message asking you to set it.

list_extractors: list all 28 site extractors

No params. Returns each extractor's name and URL shape: reddit, hackernews, github_repo, github_pr, github_issue, github_release, pypi, npm, crates_io, huggingface_model, huggingface_dataset, arxiv, docker_hub, dev_to, stackoverflow, substack_post, youtube_video, linkedin_post, instagram_post, instagram_profile, shopify_product, shopify_collection, ecommerce_product, woocommerce_product, amazon_product, ebay_listing, etsy_listing, trustpilot_reviews.

research: deep multi-source research (requires WEBCLAW_API_KEY)

query, deep, topic. Runs a search, read, and synthesize loop on the hosted engine and returns a cited report.

Which tools need a key?

Works with no key (runs locally)	Needs WEBCLAW_API_KEY (hosted)
scrape, crawl, map, batch, extract, summarize, diff, brand, vertical_scrape, list_extractors	research
search (uses your own SERPER_API_KEY; hosted fallback if unset)	escalation for bot-protected and JavaScript-rendered pages

LLM setup (for extract and summarize)

These two tools use an LLM provider chain: local Ollama first (free and private; install from ollama.com), then your own OPENAI_API_KEY, GEMINI_API_KEY, or ANTHROPIC_API_KEY if set. No webclaw key needed.

Tips

• For GitHub, Reddit, YouTube, npm, PyPI, or Amazon, use vertical_scrape (or plain scrape, which auto-detects most verticals) to get typed fields in one call.
• Set only_main_content: true to strip navigation, sidebars, and footers.
• Use the llm format when passing content to a model.
• Run map before crawl to scope a site, then crawl the section you need.
• If a page is bot-protected or JS-only, set WEBCLAW_API_KEY to escalate; without one you get a clear note that the page needs it.

vs web_fetch

	webclaw	web_fetch
Output quality	Multi-step extraction pipeline; clean markdown and llm format	Basic HTML parsing
Structured extraction	LLM- and schema-based, 28 typed extractors	None
Crawling and mapping	Whole-site crawl and URL discovery	Single page
Bot-protected and JS pages	Handled (local best-effort; automatic with a key)	Fails or readability-only
Cost	Free and local by default	Free

Use web_fetch for a quick one-off lookup. Reach for webclaw when you need reliability, clean structure, structured data, or whole-site coverage.