ai-content-discovery

AI Content Discovery

Fixes the AI Content Discovery category of IsAgentReady.com. These checkpoints test if AI systems can find, crawl and read the site.

• Methodology version: 2026-10-22. Methodology: https://isagentready.com/en/methodology
• Scoring (weights and points): https://isagentready.com/api/v2/methodology/2026-10-22/scoring
• Full checkpoint data (stage, evidence type, verify command, rescan command): references/checkpoints.md

When to Use

• The scan shows failures in AI Content Discovery (legacy ids 1.x).
• robots.txt is missing, returns HTML or blocks AI crawlers.
• The site has no sitemap, no llms.txt or no AGENTS.md.
• Agents must get Markdown, Link headers or Content-Signal preferences.
• The user asks to "make the site crawlable by AI" or to "fix AI discoverability".

When NOT to Use

• JSON-LD or Schema.org markup: use structured-data.
• Semantic HTML, headings or SSR: use content-semantics.
• MCP, A2A, OpenAPI or WebMCP: use agent-protocols.
• Security headers or OAuth metadata: use security-trust.

Instructions

1. Get the failing checkpoints from the scan. Each result has a legacy id (1.4) and a stable id (ia.llms-txt).
2. Go to the fix workflow with the anchor that is equal to the stable id.
3. Adapt the example to the site. Replace example.com with the domain of the site.
4. Run the verify command. All verify commands are read-only.
5. Rescan with isagentready results <domain> --checkpoint <stable-id> or at isagentready.com.

If $ARGUMENTS is provided, use it as the URL to fix or as the checkpoint to fix.

Checkpoints

Strength is the agentready.org spec level (MUST, SHOULD, MAY). Strength does not change the score.

Legacy	Stable ID	Name	AR IDs	Strength
1.1	ia.robots-txt	robots.txt present	AR-FIND-01	MUST
1.2	ia.ai-crawler-directives	AI crawler directives	AR-FIND-01	MUST
1.3	ia.xml-sitemap	XML sitemap	AR-FIND-03	SHOULD
1.4	ia.llms-txt	llms.txt	AR-READ-07	SHOULD
1.5	ia.meta-robots	Meta robots / X-Robots-Tag	AR-FIND-01	MUST
1.6	ia.content-freshness	Content freshness signals	none	MAY
1.7	ia.markdown-negotiation	Content negotiation (text/markdown)	AR-READ-09	MAY
1.8	ia.bot-accessibility	HTTP bot accessibility	AR-FIND-01	MUST
1.9	ia.link-headers	Link response headers	AR-FIND-04, AR-READ-06	MAY
1.10	ia.content-usage	Content-Usage preferences	none	MAY
1.11	ia.agents-md	AGENTS.md agent guide	none	MAY
1.12	ia.agent-content-contracts	Agent content contracts	AR-READ-11, AR-ACT-01	MAY
1.13	ia.soft-404	Soft-404 hygiene	AR-READ-02	MUST
1.14	ia.dns-aid	DNS-AID agent records	none	MAY
1.15	ia.pricing-page	First-party pricing page	AR-READ-11	SHOULD

Training exposure checks (te.1 to te.5) are informational and have no score. See references/checkpoints.md.

Fix Workflows

1.1 robots.txt present

Passes when /robots.txt returns 200 with Content-Type: text/plain and is not an HTML page.

1. Put a robots.txt file in the web root:

   User-agent: *
   Allow: /
   
   Sitemap: https://example.com/sitemap.xml
   

2. Serve it as text/plain. Nginx: location = /robots.txt { default_type text/plain; }.

Verify: curl -sfI https://example.com/robots.txt | grep -i '^content-type: text/plain'

1.2 AI crawler directives

Passes when robots.txt allows the AI user agents explicitly. No mention gives partial credit. A Disallow for all AI agents fails.

1. Add one group for each AI user agent. The full list is in references/robots-txt-guide.md.

   User-agent: GPTBot
   User-agent: OAI-SearchBot
   User-agent: ChatGPT-User
   User-agent: ClaudeBot
   User-agent: Claude-User
   User-agent: Claude-SearchBot
   User-agent: PerplexityBot
   User-agent: Google-Extended
   Allow: /
   

2. To block training but allow search, use Disallow: / only for the training agents.

Verify: curl -sf https://example.com/robots.txt | grep -iE '^user-agent: *(GPTBot|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot)'

1.3 XML sitemap

Passes when a Sitemap: line in robots.txt, /sitemap.xml or /sitemap_index.xml returns XML with <urlset> or <sitemapindex>.

1. Generate the sitemap with the framework (Next.js sitemap.ts, WordPress /wp-sitemap.xml, Django django.contrib.sitemaps).

   <?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url><loc>https://example.com/</loc><lastmod>2026-10-01</lastmod></url>
   </urlset>
   

2. Add Sitemap: https://example.com/sitemap.xml to robots.txt.

Verify: curl -sf https://example.com/sitemap.xml | grep -qE '<(urlset|sitemapindex)'

1.4 llms.txt

Passes when /llms.txt returns text/plain or text/markdown, starts with a # heading and has one or more URLs. A /llms-full.txt file gives a bonus.

1. Write /llms.txt:

   # Example Inc
   
   > Example Inc sells project planning software for small teams.
   
   ## Docs
   - [Getting started](https://example.com/docs/start): Set up a first project
   - [API reference](https://example.com/docs/api): REST endpoints
   

2. Serve it as text/markdown or text/plain. See references/llms-txt-guide.md.

Verify: curl -sf https://example.com/llms.txt | head -1 | grep -q '^# '

1.5 Meta robots / X-Robots-Tag

Passes when no restrictive directive is present. nofollow, nosnippet, noai or noimageai give partial credit. noindex fails.

1. Remove noindex and noai from <meta name="robots">. Use <meta name="robots" content="index, follow"> or no tag.
2. Remove restrictive X-Robots-Tag headers from the server or CDN configuration.

Verify: curl -sfI https://example.com/ | grep -i '^x-robots-tag' (expect no line, or no noindex)

1.6 Content freshness signals

Passes when JSON-LD has dateModified or the page has article:modified_time. datePublished or <time datetime> gives partial credit. A Last-Modified header alone gives less.

1. Add the dates to the JSON-LD of the page:

   { "@context": "https://schema.org", "@type": "Article",
     "datePublished": "2026-09-01T09:00:00Z", "dateModified": "2026-10-15T14:30:00Z" }
   

2. Or add <meta property="article:modified_time" content="2026-10-15T14:30:00Z">.

Verify: curl -sf https://example.com/ | grep -qE 'dateModified|article:modified_time'

1.7 Content negotiation (text/markdown)

Passes when a request with Accept: text/markdown gets Content-Type: text/markdown, a Vary: Accept header and more than 200 bytes with a heading.

1. Return a Markdown version of the page when the Accept header asks for it:

   app.get("/", (req, res) => {
     res.vary("Accept");
     if (req.accepts(["text/html", "text/markdown"]) === "text/markdown") {
       return res.type("text/markdown; charset=utf-8").send(renderMarkdown("/"));
     }
     res.send(renderHtml("/"));
   });
   

2. Make sure that the CDN keeps Vary: Accept, so that it does not cache one version for all clients.

Verify: curl -sfI -H 'Accept: text/markdown' https://example.com/ | grep -iE '^(content-type: text/markdown|vary:.*accept)'

1.8 HTTP bot accessibility

Passes when the homepage returns 2xx. A 401 or 403 from a WAF or CDN fails.

1. Find the block. Compare the status code for a browser user agent and for an AI user agent.
2. Add a WAF exception for verified AI crawlers. Cloudflare: Security, WAF, Custom rules, "User Agent contains GPTBot", action Skip.
3. Do not rate-limit the homepage for crawlers that respect robots.txt.

Verify: curl -s -o /dev/null -w '%{http_code}' -A 'Mozilla/5.0 (compatible; GPTBot/1.0)' https://example.com/

1.9 Link response headers

Passes when the homepage sends Link headers (RFC 8288) with two or more of api-catalog, service-desc, service-doc, sitemap, describedby and related. One rel gives partial credit.

1. Add the header on the homepage response:

   add_header Link '</.well-known/api-catalog>; rel="api-catalog", </sitemap.xml>; rel="sitemap"' always;
   

2. Make sure that each target URL returns 200.

Verify: curl -sfI https://example.com/ | grep -i '^link:'

1.10 Content-Usage preferences

Passes when robots.txt has a Content-Signal: line with a known key (search, ai-input, ai-train). A line without a known key gives partial credit.

1. Add the preferences to robots.txt, in the User-agent: * group:

   User-agent: *
   Content-Signal: search=yes, ai-input=yes, ai-train=no
   Allow: /
   

Verify: curl -sf https://example.com/robots.txt | grep -i '^content-signal:'

1.11 AGENTS.md agent guide

Passes when /AGENTS.md returns text, starts with a # heading and has 50 or more characters. An HTML soft-404 page fails.

1. Write /AGENTS.md:

   # Example Inc for agents
   
   - API: https://example.com/openapi.json (no key for read endpoints)
   - MCP: https://example.com/mcp (Streamable HTTP)
   - Docs: https://example.com/docs
   - Rate limit: 60 requests per minute
   

2. Serve it as text/markdown.

Verify: curl -sf https://example.com/AGENTS.md | head -1 | grep -q '^# '

1.12 Agent content contracts

Applies when the HTML shows login, pricing or app signals. Passes when /auth.md and /pricing.md (as applicable) return text with a # heading and 80 or more characters.

1. Write /pricing.md with the plans and prices as text. Write /auth.md with the sign-in method, the scopes and the agent rules.

   # Pricing
   
   - Free: 0 USD per month, 3 projects
   - Team: 12 USD per user per month, unlimited projects
   

2. Serve both files as text/markdown.

Verify: curl -sfI https://example.com/pricing.md | grep -iE '^content-type: text/(markdown|plain)'

1.13 Soft-404 hygiene

Passes when an unknown path returns 404 or 410. Other statuses or a cross-origin redirect give partial credit. A 2xx page for an unknown path fails.

1. Remove catch-all rules that return 200 or redirect to the homepage. For an SPA, let the server return 404 for unknown routes:

   location / { try_files $uri $uri/ =404; }
   

2. In a framework, return the 404 status with the not-found page. Do not render it with status 200.

Verify: curl -s -o /dev/null -w '%{http_code}' https://example.com/.well-known/isagentready-missing-check

1.14 DNS-AID agent records

Passes when _index._agents, _mcp._agents or _a2a._agents under the domain has an SVCB or HTTPS record. A TXT record only gives partial credit.

1. Add an SVCB record at the DNS provider:

   _mcp._agents.example.com. 3600 IN SVCB 1 mcp.example.com. alpn="h2" port=443
   

Verify: curl -s 'https://dns.google/resolve?name=_mcp._agents.example.com&type=SVCB' | jq -e '.Answer | length > 0'

1.15 First-party pricing page

Applies when the site shows pricing signals. Passes when /pricing (or a linked pricing page) returns 200 and shows the prices as server-rendered text. Offer JSON-LD is optional.

1. Render the plans and prices in the server HTML. Do not load them only with JavaScript.

   <section id="plans">
     <h2>Team</h2>
     <p><data value="12">$12</data> per user per month</p>
   </section>
   

2. Link the page from the homepage navigation.

Verify: curl -sf https://example.com/pricing | grep -qE '[$€£] ?[0-9]'

Key Gotchas

1. robots.txt returns the HTML 404 page with status 200.
2. robots.txt or llms.txt has Content-Type: text/html.
3. User-agent: * with Disallow: / blocks all AI crawlers.
4. The CDN drops Vary: Accept, so Markdown and HTML mix in the cache.
5. A catch-all route returns 200 for every path (soft 404).

See references/gotchas.md for correct and incorrect examples.

References

• references/checkpoints.md: generated checkpoint data for each checkpoint
• references/robots-txt-guide.md: robots.txt syntax, AI user agents and tests
• references/llms-txt-guide.md: llms.txt structure and examples for each site type
• references/gotchas.md: common errors with correct and incorrect examples