AI Web Checkby noviKEY
Menu

Technical guide · Vendor documentation

AI crawlers and robots.txt: separate search, training, and user retrieval

How to distinguish crawl policy, search visibility, model-development controls, and user-triggered retrieval instead of treating every AI user-agent token as the same mechanism.

Published
Updated

Scope: this guide explains a technical signal observed by AI Web Check. It does not claim that the signal causes ranking, indexing, citation or inclusion in AI answers.

robots.txt controls crawling, not every downstream outcome

robots.txt is published at the origin root and tells compliant automated clients which paths they may fetch. It is not a universal indexing-removal mechanism; page-level indexing directives such as noindex serve a different purpose.

AI-related vendors increasingly expose several user-agent tokens with different roles. Search discovery, model-development collection, and user-triggered retrieval should therefore be treated as separate policy decisions rather than one generic “AI bot” switch.

Current vendor roles matter

OpenAI documents OAI-SearchBot for ChatGPT search discovery and GPTBot as a control for content that publishers want excluded from potential training. Anthropic documents ClaudeBot for model-development collection, Claude-SearchBot for search-result quality, and Claude-User for user-initiated retrieval.

Google documents Googlebot as the crawler control for Google Search, including AI features in Search. Google-Extended is a separate robots.txt product token for certain Gemini training and grounding uses; Google states that it does not affect inclusion or ranking in Google Search.

What AI Web Check evaluates

AI Web Check parses robots.txt and evaluates the crawler tokens that belong to the current scored Methodology set. The result means only that the published robots policy does not create an explicit block for the evaluated token.

That status does not prove that a vendor will crawl the site, index it, quote it, train on it, or select it for an answer. Network access can also be affected by CDN, WAF, authentication, and server behavior outside robots.txt.

A deliberate policy example

Define each token according to the product behavior you actually want and re-check vendor documentation when your policy changes. Do not copy an allowlist mechanically because token roles are not interchangeable.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /private-training-excluded/

Sitemap: https://example.com/sitemap.xml

Common mistakes and verification

  • A broad Disallow unintentionally covers public content that should remain discoverable.
  • robots.txt allows a crawler while the CDN or WAF returns a challenge or 403 response.
  • Search, training, and user-triggered fetch tokens are treated as if they have the same purpose.
  • robots.txt is used as a substitute for noindex.
  • Token policy is copied from an old article without checking the vendor’s current documentation.