Canonical URLs, robots meta, and X-Robots-Tag

Three mechanisms solve different problems

rel=canonical identifies a preferred URL among duplicate or closely related resources. The robots meta tag publishes page-specific indexing and serving directives for HTML. X-Robots-Tag carries equivalent directives in HTTP headers and can also apply to non-HTML resources.

None of these mechanisms is interchangeable with robots.txt. If a crawler cannot fetch a URL because crawling is blocked, it may never see a noindex directive published in the page response.

What AI Web Check evaluates

AI Web Check checks the final page for an absolute canonical URL, robots meta directives, and X-Robots-Tag. An explicit noindex or a canonical that points to a technical or error destination can create a direct contradiction with the expected AI Readiness context.

The checker does not predict which canonical a search engine will ultimately choose. It reports what the site publishes and highlights clear conflicts between those signals.

A consistent indexable page

If a resource should genuinely be excluded from indexing, publish noindex deliberately and make sure the crawler can retrieve the response that contains it. For PDFs and other non-HTML resources, X-Robots-Tag can provide the directive at the HTTP layer.

<link rel="canonical" href="https://example.com/article">
<meta name="robots" content="index,follow,max-image-preview:large">

# HTTP response when no separate header directive is needed:
# X-Robots-Tag is absent

Typical conflicts

• canonical points to a URL that errors or permanently redirects again.
• One layer says index while another says noindex.
• robots.txt blocks the page while the team expects a crawler to read its noindex directive.
• The sitemap contains a non-canonical variant while rel=canonical points elsewhere.

Implementation and verification

Give each public page one intended canonical outcome: an indexable canonical resource, a redirect to another resource, or an intentionally noindex page. Then align redirects, canonical, sitemap, robots meta, and X-Robots-Tag around that decision.

Validate the final HTTP response and rendered source after deployment. A technically valid directive can still be wrong if it describes a different URL lifecycle from the one the server actually implements.