mcpField
S
seo-crawler-mcp

seo-crawler-mcp

Local
github.com/houtini-aiBusiness & Marketing16Updated 12w ago
Repository

About seo-crawler-mcp

seo-crawler-mcp is a retired website crawler that stored crawl data in SQLite and provided Claude with analysis queries. The repository is archived, and its crawler has been incorporated into SEO Audit Console, which combines crawl data with search data.

Key features

  1. 01Crawl websites into SQLite
  2. 02Query crawl results for analysis
  3. 03Archived with no further releases

Use cases

  1. 01Review stored crawl data with analysis queries
  2. 02Move to SEO Audit Console for continued development

MCP support

The retired project provided Claude with 25+ analysis queries over crawl data. Its crawler was rewritten and incorporated into SEO Audit Console.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config
{
  "mcpServers": {
    "houtini-ai-seo-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "@houtini/seo-crawler-mcp"
      ],
      "env": {
        "OUTPUT_DIR": "<OUTPUT_DIR>",
        "DEBUG": "<DEBUG>"
      }
    }
  }
}
Run commandnpx -y @houtini/seo-crawler-mcp
Environment variables to set
OUTPUT_DIRDirectory where crawl results are saved
DEBUGEnable verbose debug logging (set to 'true' to enable)

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages
npm@houtini/seo-crawler-mcp2.0.1

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

From the project README
SEO Audit Console

SEO Crawler MCP - retired

This project has been superseded by SEO Audit Console.

SEO Crawler crawled your whole site into SQLite and gave Claude 25+ analysis queries over the result. The crawler survived the move - it was rewritten to be stdio-safe and it is now the crawl half of a bigger tool.

SEO Audit Console keeps the crawl and adds the thing the crawl could never tell you on its own: what people actually searched for to get there. It merges the crawl with your Google Search Console history and on-demand DataForSEO data, then ranks every finding by the clicks it could recover rather than by severity.

A flat crawler sells you severity. This ranks by yield.

Where to go

git clone https://github.com/houtini-ai/seo-audit.git

Docs and setup: github.com/houtini-ai/seo-audit

What this means for you

  • The npm package still installs. @houtini/seo-crawler-mcp is deprecated, not unpublished. Nothing you have running will break.
  • This repository is archived. The code stays readable and every existing link keeps working, but there are no further releases, and issues and pull requests are closed.
  • The crawler got better on the way over. It no longer depends on Crawlee (which logged to stdout and corrupted MCP's JSON-RPC framing), it seeds discovery from sitemaps and known GSC URLs rather than links alone, and it guards against redirects that wander off-host.

Migrating

Start a crawl on the new tool and it builds its own database:

start_crawl for https://example.com/

Run refresh_property instead if you want the crawl, the Search Console sync and URL inspection in one pass.

Part of the Houtini open-source MCP set. Questions: [email protected].

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.