Scanner transparency
How AI Web Check requests your website
AI Web Check makes a limited number of requests to public website resources. This page explains what the service requests, how redirects are handled, and how requests are kept safe.
- Updated
Scanner identity
Requests identify the service with this User-Agent:
AI-Web-Check/1.1 (+https://ai.novikey.com/bot)AI Web Check is a noviKEY project. Technical or abuse reports: a@novikey.com.
What a scan requests
The base scan uses up to 5 public resources: the homepage, /robots.txt, /sitemap.xml, /llms.txt and /llms-full.txt. If the homepage redirects, these auxiliary resources are requested from the final public website address.
If the site is identified as an online store, the check may request a small number of additional HTTPS resources. Product pages and AI-agent documents stay on the same website; redirects to another website are rejected.
Request safety
- The service accepts normal HTTP and HTTPS addresses without embedded credentials.
- Requests to local, private and special-purpose network addresses are blocked.
- Every redirect destination is checked again before another request is made.
- The scanner does not use proxy settings from the server environment.
Load limits
Each resource has limits on response time, response size and redirects. The check stops when a limit is exceeded. Cookies are not sent, and HTTPS certificate verification remains enabled.
robots.txt and scan opt-out
AI Web Check requests /robots.txt first. To stop the scan, publish:
User-agent: AI-Web-Check\nDisallow: /This exact service-specific rule stops the scan before the homepage and other diagnostic resources are requested. User-agent: * is not treated as an AI Web Check opt-out.
Data handling
Fetched responses are used only for the requested check; the source HTML of the checked website is not stored in reports. See Privacy for report, cache and aggregate retention details.