seo-ai-discovery

seo-ai-discovery (M21 — AI discovery & agent endpoints)

Discovery files tell agents and coding assistants where to look; none of them is a documented answer-engine ranking or citation signal, so every M21 finding is speculative or directional and the module's weight is 0. Robots/UA posture lives in seo-ai-crawlers (M14); agentic-commerce readiness (UCP + catalog feeds on stores) lives in seo-ecommerce (M18).

Inputs

Work from the PageSnapshot named in your dispatch envelope: <run_dir>/site/discovery.json holds the probe results (status, content-type, bytes, raw bodies ≤256 KB) for /llms.txt, /llms-full.txt, /agents.md, /.well-known/ucp, /.well-known/ai-catalog.json, /sitemap_agentic_discovery.xml, /api/ucp/mcp; <run_dir>/site/robots.json holds the per-bot verdicts for the robots cross-check; vertical in the envelope decides UCP ownership. Deterministic findings already emitted by audit.mjs are listed in <run_dir>/findings.deterministic.json — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs" and use the printed snapshot path.

Audits

1. llms.txt — present (200, text); llmstxt.org structure: one H1 title, optional summary blockquote, H2 sections of - [title](url): note links; linked URLs resolve (--deep, ≤25 checked); the file is not disallowed in robots to the AI bots that would read it. /llms-full.txt is reported present/absent only.
2. agents.md — present/absent; note when it is a platform default (Shopify ships one) — informational.
3. UCP profile (/.well-known/ucp) — only when ecommerce is not active (M18 owns it otherwise): public (200, no auth), JSON, ucp.version as YYYY-MM-DD, services/capabilities objects keyed by reverse-DNS names → arrays of {version, spec, transport?, endpoint?, schema}. Note a declared dev.shopify.catalog entry — it says the platform exposes a catalog service, which is a platform default, not a merchant decision.
4. ARD catalog (/.well-known/ai-catalog.json) — present and parseable JSON. Report-only: v0.9 draft (2026-05-28), near-zero adoption.
5. Agentic sitemap (/sitemap_agentic_discovery.xml) — present, well-formed XML, sampled <loc> entries resolve.

Fixes

• AUTO (fixable: auto), disclosure-gated and scored 0: generate or repair llms.txt / llms-full.txt from the site's own structure (sitemap URLs + parsed titles) only on explicit request (fix --category ai-discovery, alias llms), shown as a diff before writing, always with the disclosure that impact is low/uncertain.
• ADVISORY (fixable: advisory): UCP/ARD/agentic-sitemap adoption, and unblocking a disallowed discovery file (the robots write belongs to M14). The tool never authors a merchant profile — platforms generate those.
• Never fabricate link targets, descriptions, endpoints, or versions — ask the user or leave a clearly-marked TODO placeholder.

Verification

• node "${CLAUDE_PLUGIN_ROOT}/scripts/ai-discovery.mjs" --snapshot <pages/<slug>.json> [--deep] — re-reads site/discovery.json, validates llms.txt structure (dom_assert), UCP/ARD JSON shape (schema_validator), agentic-sitemap XML (xml_parse), and the robots cross-check (robots_parse).
• When the discovery probes did not run (no network, local path), status is needs_api — never a false pass.

Findings

Conform to schema/finding.schema.json; axis ai; severity ≤2; confidence speculative unless noted. Severity policy: 5 is reserved for catastrophic, eligibility-killing facts at site/template scope; 4 major · 3 moderate · 2 minor · 1 cosmetic · 0 informational — and only an established severity-5 fail in an active category can cap a score (references/scoring-model.md). M21's category weight is 0, so nothing here can move or cap either score whatever its severity. evidence.observed quotes the file/probe line verbatim; verification.reproduce is the command above; expected_impact is banded (no naked %). Ids:

• M21.llmstxt.missing (warn 1, fixable: auto) · M21.llmstxt.malformed (warn 1, fixable: auto) · M21.llmstxt.broken_links (warn 2, fixable: proposed, directional) · M21.llmstxt.blocked_for_ai_bots (warn 2, fixable: advisory, directional — file exists but robots disallows it to the bots meant to read it).
• M21.agents_md.present (pass 1) · M21.agents_md.missing (warn 1, fixable: advisory).
• M21.ucp.present_valid (pass 2) · M21.ucp.invalid (warn 2, fixable: advisory) · M21.ucp.not_public (warn 2, fixable: advisory) — only when ecommerce is inactive; otherwise not_applicable naming M18 as owner.
• M21.ai_catalog.present (pass 1) · M21.ai_catalog.invalid_json (warn 1, fixable: advisory).
• M21.agentic_sitemap.present (pass 1) · M21.agentic_sitemap.broken_locs (warn 2, fixable: proposed, directional).

Honesty

• Google Search ignores llms.txt — Google's statement (2026-06-15 — TO-VERIFY the exact date; it appears in Optimizing your website for generative AI features, updated 2026-07-10, and AI features and your website says "no … AI text files"). Never present llms.txt as an AI Overviews / AI Mode lever.
• Vendor support is partial (Anthropic and Perplexity read it in retrieval; OpenAI uncommitted); adoption is ~10% and, per Ahrefs (May 2026), 97% of llms.txt files received zero bot requests. Quote that figure only in rationale with the citation — never in expected_impact.
• UCP, ARD, and the agentic sitemap concern agent commerce and resource discovery, not answer-engine citation. Report them; nothing here moves the AI Visibility score.
• Shopify stores ship all of these by default (May 2026) — their presence on a Shopify store is a platform default, not a merchant merit signal.