mcpField
M
mcp-server-spider: A spider MCP server

mcp-server-spider: A spider MCP server

Local
github.com/GeorgeLSBrowser Automation1Updated 79w ago
Repository

About mcp-server-spider: A spider MCP server

mcp-server-spider connects Spider crawling and scraping capabilities to MCP clients. It can traverse web pages and return discovered URLs or page content with links, and is in early development.

Key features

  1. 01Crawl pages and discover URLs
  2. 02Scrape page content and links
  3. 03Control traversal depth
  4. 04Filter URLs with allowlists and blocklists
  5. 05Configure request headers and user agent

Use cases

  1. 01Discover pages linked from a URL
  2. 02Collect page content and links

MCP support

Exposes crawl and scrape tools to MCP clients, returning discovered URLs or JSON data containing page URLs, links, and content.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config
{
  "mcpServers": {
    "spider": {
      "command": "uvx",
      "args": [
        "mcp-server-spider"
      ]
    }
  }
}
Run commanduvx mcp-server-spider

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages
pypimcp-server-spider

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

From the project README

mcp-server-spider: A spider MCP server

Overview

A Model Context Protocol server for Spider crawler interaction and automation. This server provides tools to crawl and scrape web pages.

Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future.

Tools
  1. crawl
    • Crawls the given url and returns the list of URLs that were found
    • Input:
      • url: The url to crawl
      • headers: Additional headers passed along with crawl requests
      • user_agent: User agent to use for the crawl requests
      • depth: The depth of link traversal
      • blacklist: A list of regural expression to blacklist URLs from the crawling process
      • whitelist: A list of regular expression to whitelist URLS from the crawling process
      • respect_robots_txt: Whether to respect robots.txt file
      • accept_invalid_certs: Whether to accept invalid certifcates or not
    • Returns: List of URLs found
  2. scrape
    • Scrapes the given url and returns a list of JSON objects that contain the url, links and content of each page discovered
    • Input: Same as crawl
    • Returns: A list of JSON objects (as a string) that contain the url, links and content of each page discovered

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will use uvx to directly run mcp-server-spider.

Using PIP

Alternatively you can install mcp-server-spider via pip:

pip install mcp-server-spider

After installation, you can run it as a script using:

python -m mcp_server_spider

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.