mcpField
W
Web Scraping API

Web Scraping API

Local
github.com/zeeb0ttBrowser Automation2Updated 74w ago
Repository

About Web Scraping API

Web Scraping API connects web-scraping endpoints to AI agent workflows for developers automating web data collection. Agents can scrape pages, extract links, paginate, and search.

Key features

  1. 01Call web-scraping endpoints
  2. 02Extract links from pages
  3. 03Fetch subsequent pages
  4. 04Search web data

Use cases

  1. 01Scrape pages for property details
  2. 02Paginate through recently listed properties
  3. 03Automate web data retrieval in AI agent workflows

MCP support

Provides an MCP server configuration that lets MCP clients connect to the scraping API and call its /scrape, /links, /next, and /search endpoints.

#web-scraping#api#ai-agents

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config
{
  "mcpServers": {
    "web-scraping-api": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote"
      ],
      "env": {
        "AUTH_HEADER": "<AUTH_HEADER>"
      }
    }
  }
}
Run commandnpx -y mcp-remote
Environment variables to set
AUTH_HEADERSecret

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages
npmmcp-remote

Tools

4

Saved tool list · updated 3w ago

ParameterTypeRequiredDescription
urlstringYes
fieldsobjectYes

Details

From the project README

🚀 We’ve just launched our new MCP server!

If you're building or using AI agents, you can now plug our web scraping API directly into your autonomous workflows. Just add the following to your MCP config:

{
  "mcpServers": {
    "web-scraping-api-by-instantapi-ai": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://web-scraping-api-by-instantapi-ai.help-052.workers.dev/sse",
        "--header",
        "Authorization:${AUTH_HEADER}"
      ],
      "env": {
        "AUTH_HEADER": "Bearer <YOUR_API_KEY>"
      }
    }
  }
}

🔐 Just replace <YOUR_API_KEY> with your API key from InstantAPI.ai. Sign up for your own API key here: https://web.instantapi.ai/#pricing-03-254921

Once connected, your AI agent can automatically invoke any of our scraping endpoints—/scrape, /links, /next, /search—to complete complex web data tasks, without needing any HTML selectors or manual logic.

🧠 Example real-world task from a user:

Go to https://www.domain.com.au/sale/bowral-nsw-2576/?bedrooms=1-any&bathrooms=1-any&excludeunderoffer=1&establishedtype=established&sort=dateupdated-desc
Extract each property link.
For each, get the full physical address and how many days since it was listed. If not known, assume 0 (today).
Keep paginating until no properties are listed in the last 7 days.
Return all matching addresses.

✅ The agent completed the task autonomously—scraping new leads for a mortgage broker with zero code written.

If you’re building AI workflows, this massively cuts down on glue code and edge case handling. Just give your agent a goal and let it fetch the data.

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.