mcpField
C
Claude Web Scraper MCP

Claude Web Scraper MCP

Local
github.com/vishwajeetdabholkarAI & Agents4Updated 81w ago
Repository

About Claude Web Scraper MCP

Claude Web Scraper MCP connects Claude for Desktop to a locally running eGet web scraper. It is intended for people who want Claude to retrieve website content for summarization and analysis in conversations.

Key features

  1. 01Connect to the local eGet API
  2. 02Scrape website content
  3. 03Use scraped content in Claude

Use cases

  1. 01Scrape a page and summarize it
  2. 02Gather information from a website
  3. 03Analyze scraped web content in a conversation

MCP support

The server connects Claude for Desktop to the local eGet scraper API and makes website scraping available through MCP.

#web-scraper#mcp#claude

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

No install method listed yet

We haven’t collected a package or remote endpoint for this server. Follow the instructions in the project README.

Repository

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

From the project README

Claude Web Scraper MCP

A simple Model Context Protocol (MCP) server that connects Claude for Desktop to a locally running eGet web scraper. This allows Claude to scrape website content through your local API.

Prerequisites

Setup Instructions

1. Set up eGet Web Scraper

First, make sure you have the eGet web scraper running:

# Clone the eGet repository
git clone https://github.com/vishwajeetdabholkar/eGet-Crawler-for-ai
cd eGet-Crawler-for-ai

# Set up and run eGet according to its instructions
# (typically using Docker or local Python installation)

# Verify the API is running (default: http://localhost:8000/api/v1/scrape)
2. Set up the MCP Server
# Create project directory
mkdir claude-scraper-mcp
cd claude-scraper-mcp

# Set up UV and virtual environment
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
uv add "mcp[cli]" httpx

# Create the MCP server script
touch scrape_mcp_server.py

Copy the scrape_mcp_server.py code into the file.

3. Configure Claude for Desktop
  1. Create or edit the Claude desktop configuration:
# On macOS
mkdir -p ~/Library/Application\ Support/Claude/
  1. Add this configuration to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
    "mcpServers": {
        "scrape-service": {
            "command": "/absolute/path/to/claude-scraper-mcp/.venv/bin/python",
            "args": [
                "/absolute/path/to/claude-scraper-mcp/scrape_mcp_server.py"
            ]
        }
    }
}

Replace the paths with the actual absolute paths to your virtual environment and script.

  1. Restart Claude for Desktop

Usage

Once set up, you can use Claude to scrape websites with commands like:

Troubleshooting

If you encounter issues:

  1. Check that eGet scraper is running
  2. Verify the API endpoint in the script matches your eGet configuration
  3. Make sure Claude for Desktop is using the correct Python interpreter
  4. Restart Claude for Desktop after making changes to the configuration

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.