mcpField
R
RAG Documentation Scraper

RAG Documentation Scraper

Local
github.com/LernowaAdminBrowser Automation
Repository

About RAG Documentation Scraper

RAG Documentation Scraper crawls documentation sites and turns their main content into metadata-rich chunks for RAG pipelines. It supports several chunking and output formats, with optional incremental crawling and an MCP wrapper for AI clients.

Key features

  1. 01Crawls documentation with Playwright
  2. 02Extracts clean main-page content
  3. 03Splits text with configurable strategies
  4. 04Exports RAG and framework-specific formats
  5. 05Tracks changes and deduplicates chunks
  6. 06Supports PDF text and OCR extraction

Use cases

  1. 01Preparing documentation for vector database ingestion
  2. 02Refreshing a documentation corpus while skipping unchanged pages
  3. 03Extracting a selected version of documentation
  4. 04Calling documentation scraping from an MCP client

MCP support

An included MCP server exposes the `scrape_documentation_for_rag` tool, which crawls a documentation URL and returns chunked text with source metadata. It supports local stdio and HTTP with SSE transports.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config
{
  "mcpServers": {
    "lernowaadmin-rag-docs-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "rag-docs-scraper-mcp-server"
      ],
      "env": {
        "APIFY_TOKEN": "<APIFY_TOKEN>"
      }
    }
  }
}
Run commandnpx -y rag-docs-scraper-mcp-server
Environment variables to set
APIFY_TOKENSecretYour Apify API token — https://console.apify.com/account/integrations

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages
npmrag-docs-scraper-mcp-server1.0.0

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

The maintainer has not added details yet. Check the README in the repository.

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.