AI Web Checkby noviKEY
Menu

Scanner transparency

How AI Web Check requests your website

AI Web Check makes a limited number of requests to public website resources. This page explains what the service requests, how redirects are handled, and how requests are kept safe.

Updated

Scanner identity

Requests identify the service with this User-Agent:

AI-Web-Check/1.1 (+https://ai.novikey.com/bot)

AI Web Check is a noviKEY project. Technical or abuse reports: a@novikey.com.

What a scan requests

The base check uses up to 6 logical resource operations: /robots.txt first, the homepage, /sitemap.xml, /llms.txt, /llms-full.txt, and one read-only POST /mcp/ using server/discover for MCP 2026-07-28.

If Remote MCP advertises a tools capability, the checker may make one additional read-only tools/list request. Advertised tools are never executed.

Readiness sampling may use up to 3 additional resource operations to inspect up to 2 internal sitemap pages. A confirmed online-store context may use up to 5 more HTTPS resource operations for product samples and public agent-discovery documents.

The scan uses up to 15 logical resource operations, not a total of 15 HTTP requests. Each GET operation may follow up to 3 redirects, requiring additional HTTP requests. Time and response-size limits still apply. Remote MCP POST requests do not follow redirects.

Request safety

  • The service accepts normal HTTP and HTTPS addresses without embedded credentials.
  • Requests to local, private and special-purpose network addresses are blocked.
  • Every redirect destination is checked again before another request is made.
  • The scanner does not use proxy settings from the server environment.

Load limits

Each resource has limits on response time, response size and redirects. The check stops when a limit is exceeded. Cookies are not sent, and HTTPS certificate verification remains enabled.

robots.txt and scan opt-out

AI Web Check requests /robots.txt first. To stop the scan before the homepage and other diagnostic resources are requested, publish:

User-agent: AI-Web-Check
Disallow: /

This exact service-specific rule is treated as an AI Web Check opt-out. User-agent: * is not treated as a service-specific refusal.

Data handling

Fetched responses are used only for the requested check; the source HTML of the checked website is not stored in reports. See Privacy for report, cache and aggregate retention details.