mult-fetch-mcp-server

mult-fetch-mcp-server

This project implements an MCP-compliant client and server for communication between AI assistants and external tools.

English | 中文文档

Version 2.0

Call the fetch tool with a URL. The default format is Markdown, using an HTTP fetch plus readable-article extraction. startCursor is not required.

format	What you get
markdown	Readable article as Markdown (default)
plaintext	Readable article as plain text
html	HTML markup
json	JSON document
txt	Raw response body

A normal page comes back in one response. The default budget is maxChars 100000 and maxBytes 1048576, and a cut prefers a newline or a space. When the body is longer, the response includes nextCursor and chunkId. Call fetch again with cursor set to nextCursor.

fetch_html, fetch_json, fetch_txt, fetch_markdown, and fetch_plaintext are thin aliases of fetch with a fixed format.

Cursor

Add this to ~/.cursor/mcp.json, or to .cursor/mcp.json in a project:

{
  "mcpServers": {
    "mult-fetch": {
      "command": "npx",
      "args": ["-y", "@lmcc-dev/mult-fetch-mcp-server"],
      "env": {
        "MCP_LANG": "en"
      }
    }
  }
}

Claude Desktop

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "mult-fetch": {
      "command": "npx",
      "args": ["-y", "@lmcc-dev/mult-fetch-mcp-server"],
      "env": {
        "MCP_LANG": "en"
      }
    }
  }
}

Restart the host after saving. Ask it to fetch a URL. It should call fetch without startCursor.

Browser mode

The default HTTP fetcher does not need Chrome or Puppeteer. A normal install works without them (npm install --omit=optional skips the optional packages). To render pages in a headless browser, install the optional packages and set useBrowser to true:

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

autoDetectMode defaults to false, so a 403 does not launch a browser unless you set that flag.

Security

Every request, including redirects, is checked before it is sent:

• Loopback (127.0.0.0/8, ::1, localhost)
• Private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
• Link-local and cloud metadata (169.254.0.0/16, including 169.254.169.254, and IPv6 fe80::/10)
• Carrier-grade NAT 100.64.0.0/10, documentation ranges, and private addresses embedded in 6to4, NAT64, or IPv4-mapped IPv6

Direct connections are pinned to the addresses that passed that check. The default timeout is 30 seconds, and each response is limited by maxChars and maxBytes. Browser mode keeps the same URL checks. Set ALLOW_PRIVATE_NETWORK_REQUESTS=true only when you intentionally want the browser to open a private-network target.

Project Structure

fetch-mcp/
├── src/                         # Source code directory
│   ├── lib/                     # Library files
│   │   ├── fetchers/            # Web fetching implementation
│   │   │   ├── browser/         # Browser-based fetching
│   │   │   │   ├── BrowserFetcher.ts      # Browser fetcher implementation
│   │   │   │   ├── BrowserInstance.ts     # Browser instance management
│   │   │   │   └── PageOperations.ts      # Page interaction operations
│   │   │   ├── node/            # Node.js-based fetching
│   │   │   └── common/          # Shared fetching utilities
│   │   ├── utils/               # Utility modules
│   │   │   ├── ChunkManager.ts        # Content chunking
│   │   │   ├── ContentProcessor.ts    # HTML to text conversion
│   │   │   ├── ContentExtractor.ts    # Intelligent content extraction
│   │   │   ├── ContentSizeManager.ts  # Content size limiting
│   │   │   └── ErrorHandler.ts        # Error handling
│   │   ├── server/              # Server-related modules
│   │   │   ├── index.ts         # Server entry
│   │   │   ├── browser.ts       # Browser management
│   │   │   ├── fetcher.ts       # Web fetching logic
│   │   │   ├── tools.ts         # Tool registration and handling
│   │   │   ├── resources.ts     # Resource handling
│   │   │   ├── prompts.ts       # Prompt templates
│   │   │   └── types.ts         # Server type definitions
│   │   ├── i18n/                # Internationalization support
│   │   └── types.ts             # Common type definitions
│   ├── client.ts                # MCP client implementation
│   └── mcp-server.ts            # MCP server main entry
├── index.ts                     # Server entry point
├── tests/                       # Test files
└── dist/                        # Compiled files

MCP Specification

The Model Context Protocol (MCP) defines two main transport methods:

1. Standard Input/Output (Stdio): The client starts the MCP server as a child process, and they communicate through standard input (stdin) and standard output (stdout).
2. Server-Sent Events (SSE): Used to pass messages between client and server.

This project implements the Standard Input/Output (Stdio) transport method.

Features

• Implementation based on the official MCP SDK
• Support for Standard Input/Output (Stdio) transport
• One fetch tool with format set to HTML, JSON, text, Markdown, or plain text
• HTTP fetch by default, with optional headless-browser rendering
• Single-response content window (maxChars / maxBytes) and optional cursor continuation when the body is truncated
• Detailed debug logging to stderr
• Bilingual internationalization (English and Chinese)
• Modular design for easy maintenance and extension
• Intelligent Content Extraction: Based on Mozilla's Readability library, capable of extracting meaningful content from web pages while filtering out advertisements and navigation elements
• Metadata Support: Ability to extract webpage metadata such as title, author, publication date, and site information
• Smart Content Detection: Automatically detects if a page contains meaningful content, filtering out login pages, error pages, and other pages without substantial content
• Browser Automation Enhancements: Support for page scrolling, cookie management, selector waiting, and other advanced browser interactions

Installation

Installing via Smithery

To install Mult Fetch MCP Server for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @lmcc-dev/mult-fetch-mcp-server --client claude

Local Installation

pnpm install

Global Installation

pnpm add -g @lmcc-dev/mult-fetch-mcp-server

Or run directly with npx (no installation required):

npx @lmcc-dev/mult-fetch-mcp-server

Integration with Claude

To integrate this tool with Claude desktop, you need to add server configuration:

Configuration File Location

• MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%/Claude/claude_desktop_config.json

Configuration Examples

Method 1: Using npx (Recommended)

This method is the simplest, doesn't require specifying the full path, and is suitable for global installation or direct use with npx:

{
  "mcpServers": {
    "mult-fetch-mcp-server": {
      "command": "npx",
      "args": ["@lmcc-dev/mult-fetch-mcp-server"],
      "env": {
        "MCP_LANG": "en"  // Set language to English, options: "zh" or "en"
      }
    }
  }
}

Method 2: Specifying Full Path

If you need to use a specific installation location, you can specify the full path:

{
  "mcpServers": {
    "mult-fetch-mcp-server": {
      "command": "path-to/bin/node",
      "args": ["path-to/@lmcc-dev/mult-fetch-mcp-server/dist/index.js"],
      "env": {
        "MCP_LANG": "en"  // Set language to English, options: "zh" or "en"
      }
    }
  }
}

Please replace path-to/bin/node with the path to the Node.js executable on your system, and replace path-to/@lmcc-dev/mult-fetch-mcp-server with the actual path to this project.

Usage Examples

Below is an example of using this tool in Claude desktop client:

The image shows how Claude can use the fetch tools to retrieve web content and process it according to your instructions.

Usage

After configuration, restart Claude desktop, and you can use the following tools in your conversation:

• fetch: Fetch a URL. Set format to html, markdown (default), plaintext, json, or txt
• fetch_html, fetch_json, fetch_txt, fetch_markdown, fetch_plaintext: aliases of fetch kept from 1.x

Build

pnpm run build

Run Server

pnpm run server
# or
node dist/index.js
# if globally installed, you can run directly
@lmcc-dev/mult-fetch-mcp-server
# or use npx
npx @lmcc-dev/mult-fetch-mcp-server

Client Demo Tools

Note: The following client.js functionality is provided for demonstration and testing purposes only. When used with Claude or other AI assistants, the MCP server is driven by the AI, which manages the chunking process automatically.

Command Line Client

The project includes a command-line client for testing and development purposes:

pnpm run client <method> <params_json>
# example
pnpm run client fetch_html '{"url": "https://example.com", "debug": true}'

Demo Client Chunk Control Parameters

When testing with the command-line client, you can use these parameters to demonstrate content chunking capabilities:

• --all-chunks: Command line flag to automatically fetch all chunks in sequence (demonstration purpose only)
• --max-chunks: Command line flag to limit the maximum number of chunks to fetch (optional, default is 10)

Real-time Output Demo

The client.js demo tool provides real-time output capabilities:

node dist/src/client.js fetch_html '{"url":"https://example.com", "startCursor": 0, "contentSizeLimit": 500}' --all-chunks --debug

The demo client will automatically fetch all chunks in sequence and display them immediately, showcasing how large content can be processed in real-time.

Run Tests

# Run MCP functionality tests
npm run test:mcp

# Run mini4k.com website tests
npm run test:mini4k

# Run direct client call tests
npm run test:direct

Language Settings

This project supports Chinese and English bilingual internationalization. You can set the language using environment variables:

Using Environment Variables

Set the MCP_LANG environment variable to control the language:

# Set to English
export MCP_LANG=en
npm run server

# Set to Chinese
export MCP_LANG=zh
npm run server

# Windows system
set MCP_LANG=zh
npm run server

Using environment variables ensures that all related processes (including the MCP server) use the same language settings.

Default Language

By default, the system will choose a language according to the following priority:

1. MCP_LANG environment variable
2. Operating system language (if it starts with "zh", use Chinese)
3. English (as the final fallback option)

Debugging

This project follows the MCP protocol specification and does not output any logs by default to avoid interfering with JSON-RPC communication. Debug information is controlled through call parameters:

Using the debug Parameter

Set the debug: true parameter when calling a tool:

{
  "url": "https://example.com",
  "debug": true
}

Debug messages are sent to the standard error stream (stderr) using the following format:

[MCP-SERVER] MCP server starting...
[CLIENT] Fetching URL: https://example.com

Debug Log File

When debug mode is enabled, all debug messages are also written to a log file located at:

~/.mult-fetch-mcp-server/debug.log

This log file can be accessed through the MCP resources API:

// Access the debug log file
const result = await client.readResource({ uri: "file:///logs/debug" });
console.log(result.contents[0].text);

// Clear the debug log file
const clearResult = await client.readResource({ uri: "file:///logs/clear" });
console.log(clearResult.contents[0].text);

Proxy Settings

This tool supports various methods to configure proxy settings:

1. Using the proxy Parameter

The most direct way is to specify the proxy in the request parameters:

{
  "url": "https://example.com",
  "proxy": "http://your-proxy-server:port",
  "debug": true
}

2. Using Environment Variables

The tool will automatically detect and use proxy settings from standard environment variables:

# Set proxy environment variables
export HTTP_PROXY=http://your-proxy-server:port
export HTTPS_PROXY=http://your-proxy-server:port

# Run the server
npm run server

3. System Proxy Detection

The tool attempts to detect system proxy settings based on your operating system:

• Windows: Reads proxy settings from environment variables using the set command
• macOS/Linux: Reads proxy settings from environment variables using the env command

4. Proxy Troubleshooting

If you're having issues with proxy detection:

1. Use the debug: true parameter to see detailed logs about proxy detection
2. Explicitly specify the proxy using the proxy parameter
3. Ensure your proxy URL is in the correct format: http://host:port or https://host:port
4. For websites that require browser capabilities, set useBrowser: true to use browser mode

5. Browser Mode and Proxies

When using browser mode (useBrowser: true), the tool will:

1. First try to use the explicitly specified proxy (if provided)
2. Then try to use system proxy settings
3. Finally, proceed without a proxy if none is found

Browser mode is particularly useful for websites that implement anti-scraping measures or require JavaScript execution.

Parameter Handling

This project handles parameters in the following ways:

• debug: Passed through call parameters, each request can individually control whether to enable debug output
• MCP_LANG: Retrieved from environment variables, controls the language settings of the entire server

Usage

Creating a Client

import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
import path from 'path';
import { fileURLToPath } from 'url';

// Get the directory path of the current file
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

// Create client transport layer
const transport = new StdioClientTransport({
  command: 'node',
  args: [path.resolve(__dirname, 'dist/index.js')],
  stderr: 'inherit',
  env: {
    ...process.env  // Pass all environment variables, including MCP_LANG
  }
});

// Create client
const client = new Client({
  name: "example-client",
  version: "1.0.0"
});

// Connect to transport layer
await client.connect(transport);

// Use client
const result = await client.callTool({
  name: 'fetch_html',
  arguments: {
    url: 'https://example.com',
    debug: true  // Control debug output through parameters
  }
});

if (result.isError) {
  console.error('Fetch failed:', result.content[0].text);
} else {
  console.log('Fetch successful!');
  console.log('Content preview:', result.content[0].text.substring(0, 500));
}

Supported Tools

• fetch: Fetch a URL. format is html, markdown (default), plaintext, json, or txt
• fetch_html, fetch_json, fetch_txt, fetch_markdown, fetch_plaintext: 1.x aliases of fetch

Resources Support

The server includes support for the resources/list and resources/read methods, but currently no resources are defined in the implementation. The resource system is designed to provide access to project files and documentation, but this feature is not fully implemented yet.

Resource Usage Example

// Example: List available resources
const resourcesResult = await client.listResources({});
console.log('Available resources:', resourcesResult);

// Note: Currently this will return empty lists for resources and resourceTemplates

Supported Prompt Templates

The server provides the following prompt templates:

• fetch-website: Get website content, supporting different formats and browser mode
• extract-content: Extract specific content from a website, supporting CSS selectors and data type specification
• debug-fetch: Debug website fetching issues, analyze possible causes and provide solutions

Prompt Template Usage

1. Use prompts/list to get a list of available prompt templates
2. Use prompts/get to get specific prompt template content

// Example: List available prompt templates
const promptsResult = await client.listPrompts({});
console.log('Available prompts:', promptsResult);

// Example: Get website content prompt
const fetchPrompt = await client.getPrompt({
  name: "fetch-website",
  arguments: {
    url: "https://example.com",
    format: "html",
    useBrowser: "false"
  }
});
console.log('Fetch website prompt:', fetchPrompt);

// Example: Debug website fetching issues
const debugPrompt = await client.getPrompt({
  name: "debug-fetch",
  arguments: {
    url: "https://example.com",
    error: "Connection timeout"
  }
});
console.log('Debug fetch prompt:', debugPrompt);

Parameter Options

Each tool supports the following parameters:

Basic Parameters

• url: URL to fetch (required)
• headers: Custom request headers (optional, default {})
• proxy: Proxy server URL in the format http://host:port or https://host:port (optional)

Network Control Parameters

• timeout: Timeout in milliseconds (optional, default is 30000)
• maxRedirects: Maximum number of redirects to follow (optional, default is 10)
• noDelay: Whether to disable random delay between requests (optional, default is false)
• useSystemProxy: Whether to use system proxy (optional, default is true)

Content Window

• maxChars: Maximum characters in this response (optional, default 100000). The cut prefers a newline or space.
• maxBytes: Maximum UTF-8 bytes in this response (optional, default 1048576).
• cursor: Continuation token from nextCursor. Omit it on the first call. When it is set, the server does not refetch the URL.
• enableContentSplitting: When false, oversized content is truncated with no cursor (optional, default true).
• contentSizeLimit: Legacy byte-chunk size. When set without maxChars or maxBytes, the 1.x byte splitter is used.
• startCursor: Legacy byte offset. Not required. Use it only with a 1.x chunkId.

Chunk Management

• chunkId: Document id returned with nextCursor when a response was truncated.

Pass cursor back to fetch to read the next slice. startCursor is not part of that flow.

Mode Control Parameters

• useBrowser: Whether to use browser mode (optional, default is false)
• useNodeFetch: Whether to force using Node.js mode (optional, default is false, mutually exclusive with useBrowser)
• autoDetectMode: Retry a 403 or challenge page in the browser (optional, default false). Requires the optional Puppeteer packages.

Browser Mode Specific Parameters

• waitForSelector: Selector to wait for in browser mode (optional, default is 'body')
• waitForTimeout: Timeout to wait in browser mode in milliseconds (optional, default is 5000)
• scrollToBottom: Whether to scroll to the bottom of the page in browser mode (optional, default is false)
• saveCookies: Whether to save cookies in browser mode (optional, default is true)
• closeBrowser: Whether to close the browser instance (optional, default is false)

Content Extraction Parameters

• extractContent: Use Readability before conversion (optional). Default true for markdown and plaintext, false for html, json, and txt.
• includeMetadata: Whether to include metadata in the extracted content (optional, default false, only works when extractContent is true)
• fallbackToOriginal: Whether to fall back to the original content when extraction fails (optional, default true, only works when extractContent is true)

Debug Parameters

• debug: Whether to enable debug output (optional, default false)

Content Extraction Feature

Use the content extraction feature to get the core content of a webpage, filtering out navigation bars, advertisements, sidebars, and other distracting elements:

{
  "url": "https://example.com/article",
  "extractContent": true,
  "includeMetadata": true
}

The extracted content will include the following metadata (if available):

• Title
• Byline (author)
• Site name
• Excerpt
• Content length
• Readability flag (isReaderable)

Special Usage

Content Extraction Examples

To extract only the meaningful content from an article webpage:

{
  "url": "https://example.com/news/article",
  "extractContent": true,
  "includeMetadata": true
}

For websites where content extraction might fail, you can use fallbackToOriginal to ensure you get some content:

{
  "url": "https://example.com/complex-layout",
  "extractContent": true,
  "fallbackToOriginal": true
}

Closing Browser Without Fetching

To close the browser instance without performing any fetch operation:

{
  "url": "about:blank",
  "closeBrowser": true
}

Proxy Priority

The proxy is determined in the following order:

1. Command line specified proxy
2. proxy parameter in the request
3. Environment variables (if useSystemProxy is true)
4. Git configuration (if useSystemProxy is true)

If proxy is set, useSystemProxy will be automatically set to false.

Debug Output

When debug: true is set, logs will be output to stderr with the following prefixes:

• [MCP-SERVER]: Logs from the MCP server
• [NODE-FETCH]: Logs from the Node.js fetcher
• [BROWSER-FETCH]: Logs from the browser fetcher
• [CLIENT]: Logs from the client
• [TOOLS]: Logs from the tool implementation
• [FETCHER]: Logs from the main fetcher interface
• [CONTENT]: Logs related to content handling
• [CONTENT-PROCESSOR]: Logs from the HTML content processor
• [CONTENT-SIZE]: Logs related to content size management
• [CHUNK-MANAGER]: Logs related to content chunking operations
• [ERROR-HANDLER]: Logs related to error handling
• [BROWSER-MANAGER]: Logs from the browser instance manager
• [CONTENT-EXTRACTOR]: Logs from the content extractor

License

MIT

---

Updated by lmcc-dev