About crawl4-mcp
crawl4-mcp is a web-scraping MCP server built with crawl4ai. It crawls web content and saves it as local Markdown files, making the captured material available for RAG workflows.
Key features
- 01Crawl web content
- 02Save crawled content as local Markdown
Use cases
- 01Collect web pages for a RAG workflow
- 02Save scraped material as Markdown for later use
MCP support
The server can run over SSE and connect to MCP clients using a configured URL.
Configuration
Local server config is the same for every client. Copy it into your client’s MCP config file.
We haven’t collected a package or remote endpoint for this server. Follow the instructions in the project README.
Tools
Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.
Details
From the project READMEcrawl4-mcp
本项目是一个crawl4ai 的爬虫MCP SERVER，提供高级网络爬虫。使用此 MCP SERVER，您可以抓取任何内容，将内容保存为本地markdown文件，然后在任何地方将该知识用于 RAG。
环境要求
- Python 3.12 或更高版本
- uv 包管理器
安装步骤
-
克隆仓库
git clone <仓库地址> cd crawl4-mcp -
使用 uv 创建虚拟环境并安装依赖
uv venv -p 3.12 .\.venv\Scripts\activate uv sync crawl4ai-setup -
根据下面的配置部分创建一个.env文件
运行
```bash
uv run src/crawl4ai_mcp.py
```
server将启动并监听配置的主机和端口。
与 MCP client集成
一旦服务器使用 SSE 传输运行，您就可以使用以下配置连接到它：
{
"mcpServers": {
"crawl4-mcp": {
"transport": "sse",
"url": "http://localhost:8051/sse"
}
}
}
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.