Agent Crawl Readiness
RemoteAbout Agent Crawl Readiness
Agent Crawl Readiness is an MCP server that checks a website's readiness for agent access by examining robots.txt, sitemap.xml, llms.txt, and homepage HTML. It is for site operators assessing how agents can crawl their sites.
Key features
- 01Checks robots.txt
- 02Checks sitemap.xml and llms.txt
- 03Inspects homepage HTML
- 04Returns a bounded readiness report
Use cases
- 01Assessing a site's agent crawl readiness
- 02Reviewing crawl-related files and homepage HTML
MCP support
The MCP server runs a bounded readiness check when called with a website and returns the report. Its stated check covers robots.txt, sitemap.xml, llms.txt, and homepage HTML.
Configuration
Remote servers need no local install. Pick your client, then copy the config or install in one click.
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcpclaude mcp add --transport http sadri-dridi-agent-crawl-readiness https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp{"mcpServers":{"sadri-dridi-agent-crawl-readiness":{"type":"http","url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}Open in Cursor{"servers":{"sadri-dridi-agent-crawl-readiness":{"type":"http","url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}Open in VS Code{"mcpServers":{"sadri-dridi-agent-crawl-readiness":{"type":"http","url":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}{"mcpServers":{"sadri-dridi-agent-crawl-readiness":{"serverUrl":"https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"}}}Run commands in a terminal; paste JSON into the client’s MCP config and restart. Servers that need sign-in will prompt on first connect.
Tools
31| Parameter | Type | Required | Description |
|---|---|---|---|
| ref | string | No | Git ref name; discarded after the shape check |
| url | string | No | HTTPS URL to normalize or cite |
| city | string | No | City name for a public weather hint; discarded after the call |
| feed | string | No | Public RSS or Atom URL; titles discarded |
| host | string | No | Public hostname |
| json | string | No | JSON text to validate; discarded after the check |
| path | string | No | File path to check; no disk access |
| zone | string | No | IANA timezone name |
| query | string | No | Search text; discarded after the length check |
| website | string | No | Public https website (homepage domain) to check for agent readiness: robots.txt, llms.txt, sitemap.xml, extractable HTML. Bounded, identified, robots-respecting. |
Details
From the project READMEAgent Crawl Readiness
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage HTML (Agent Ads). Pass website=https://domain to run the bounded check now and get the report here. 7-day trial, founding $50/mo (first 10, 12 mo) then $100/mo. Follow start_here.hop first (302). Task text discarded.
An Agent Ads door on the Agent Observatory sensor. The primary tool returns the Agent Ads index (placement, measurement, a 7-day trial) starting at the readiness hop; readiness doors also run a bounded website readiness check when called with website=<https://domain>. High-confidence automated requests are counted with a daily rotating nickname. Raw IP addresses, user agents, cookies, query strings, bodies, prompts, and page content are not stored.
- MCP: https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp
- Official name:
io.github.sadri-dridi/agent-crawl-readiness - Agent Ads: https://agent-ads-production-9f45.up.railway.app/
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.