AI Bot Access Checker

AI Bot Access Checker

Can ChatGPT, Claude, Perplexity and Google actually reach your page? This tool checks 13 AI and search crawlers against one URL and tells you, for each, whether robots.txt blocks it or whether your server or firewall refuses it while a normal browser gets the page.

• Web version: seopaslaugos.io/en/tools/ai-bot-access-checker
• REST API: https://tools.seopaslaugos.io/v1/check?url=example.com (OpenAPI)
• Remote MCP server: https://tools.seopaslaugos.io/mcp (Streamable HTTP, tool check_ai_bot_access)
• Agent skill: skills/ai-bot-access-check/SKILL.md

Free, no key, MIT licence.

Why

robots.txt is only half of it. Shared-hosting security rules (ModSecurity), rate limiters that answer 429 to bots on uncached pages, and CDN bot settings (Cloudflare's AI crawler blocking) refuse AI crawlers before robots.txt is ever read, often by default. Site owners then wonder why ChatGPT says it "cannot access" their site. Most checkers only read robots.txt; this one also requests the page as each bot and compares the answer with a browser's.

Bots checked

Bot	Vendor	Used for
GPTBot	OpenAI	Training
OAI-SearchBot	OpenAI	ChatGPT search index
ChatGPT-User	OpenAI	Live visits when a user asks (OpenAI: robots.txt may not apply)
ClaudeBot	Anthropic	Training
Claude-SearchBot	Anthropic	Claude search index
Claude-User	Anthropic	Live visits when a user asks
PerplexityBot	Perplexity	Perplexity search index
Perplexity-User	Perplexity	Live visits when a user asks (Perplexity: generally ignores robots.txt)
Googlebot	Google	Google Search, AI Overviews, AI Mode
Google-Extended	Google	Gemini training opt-out (robots.txt token only; no effect on Search)
bingbot	Microsoft	Bing search index
Applebot-Extended	Apple	Apple model training opt-out (robots.txt token only; no effect on Apple search)
CCBot	Common Crawl	Open training corpus

Use it

REST

curl "https://tools.seopaslaugos.io/v1/check?url=example.com&format=text"
curl "https://tools.seopaslaugos.io/v1/check?url=https://example.com/pricing&bots=GPTBot,ClaudeBot"

AI bot access for https://www.example.lt/
Browser baseline: HTTP 200
robots.txt: found (HTTP 200)

GPTBot             BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
OAI-SearchBot      BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
Claude-User        ALLOWED
Googlebot          ALLOWED
...

MCP (Claude Code, Cursor, ChatGPT, any MCP client)

{
  "mcpServers": {
    "ai-bot-access-checker": { "type": "http", "url": "https://tools.seopaslaugos.io/mcp" }
  }
}

Hermes Agent

hermes plugins install ai-bot-access-checker

The package is an Agent Plugins v1 bundle: plugin.json, mcp.json (remote Streamable HTTP server, no keys, no local process) and the skill in skills/. The one tool is read-only; it sends HTTP requests to the URL you ask it to check, from tools.seopaslaugos.io.

Agent skill

Copy skills/ai-bot-access-check into your agent's skills folder, or install it from ClawHub. The skill calls the REST API with curl, so it needs no MCP support.

Results

access	Meaning
allowed	robots.txt allows it and the server answers it like a browser
blocked_by_robots	A Disallow rule applies (rule and user-agent group included). For ChatGPT-User and Perplexity-User the vendors say robots.txt may be ignored; robots.vendorNote says so
blocked_by_server	robots.txt allows it, the server refuses the User-Agent: 403, 429, dropped connection, challenge or captcha page
inconclusive	The firewall refuses a fake Googlebot too, so it checks crawler IPs; the real crawler may get in
suspicious	HTTP 200 but under a fifth of the browser's response size
page_error	A normal browser fails as well

How it works and its limits

• robots.txt is parsed per RFC 9309: most specific user-agent group, merged duplicate groups, longest match wins, Allow wins ties, * and $ wildcards; a 4xx robots.txt means no restrictions, a 5xx means full disallow.
• The page is requested once with a desktop Chrome User-Agent and once per bot with the bot's published User-Agent, from Cloudflare's network.
• Requests do not come from the vendors' IP ranges. Blocks by User-Agent (most hosting firewalls) are caught; firewalls that verify crawler IPs are reported as inconclusive instead of guessed.
• Rate limits: 20 checks a minute per caller IP, and 6 fresh checks a minute per checked website from all callers together, so the checker cannot be used to flood a site. Results are cached 10 minutes per URL.

Security

• Only public http/https URLs on ports 80/443 with a domain name are checked. IP literals (dotted, decimal, octal, hex, IPv6), localhost and reserved TLDs, wildcard-DNS hosts that point at private addresses (nip.io, sslip.io, ...), URLs with credentials and the checker's own hosts are refused.
• Redirects are followed by hand (max 5) and every hop is validated the same way, so a public page cannot bounce the checker to an internal address, another port or its own API.
• At most 64 KiB of each response is read; each request times out after 10 s.
• robots.txt matching uses a linear wildcard matcher with a work budget and caps on lines, rules and pattern length, so a hostile robots.txt cannot stall the Worker; if the budget runs out the result says inconclusive.
• Text taken from the checked site (robots.txt rules, URLs, headers) is stripped of control and bidi characters and length-capped before it is returned to people or AI agents.
• MCP batches are capped at 10 messages and one tools/call; request bodies are capped at 16 KiB.
• No keys, cookies or user data are stored. Results are cached by URL only.

Found a problem? Open an issue or email [email protected].

Run your own

npm install
npm test          # robots.txt parser tests
npx wrangler dev  # local Worker on :8787

Deploy with npx wrangler deploy after changing the route in wrangler.toml.

Author

Built by Mantas Tamošaitis, SEO specialist in Lithuania, who works on Google and AI search visibility for service businesses and e-shops at seopaslaugos.io. Issues and pull requests welcome.