AI crawler access
If an AI crawler cannot fetch your pages, nothing else in this report matters. We check what each named AI crawler is allowed to do, and whether your server actually lets it in.
What we check
- Your robots.txt rules for the named AI crawlers of each operator, listed by purpose. For example: OpenAI (GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for requests made by users), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended and Applebot-Extended
- A live request to your key pages with each crawler’s user agent, to catch firewall or CDN rules that block them even when robots.txt allows them
- Page-level blocks on key pages: noindex, nosnippet and X-Robots-Tag headers
What we say plainly:
Some providers verify crawlers by IP address, which a user-agent test cannot reproduce. Where a firewall is involved, we state what we could and could not confirm.
Google-Extended controls whether your content is used for Google’s Gemini models. Google states that it does not affect Google Search, including AI Overviews, so we do not treat it as an AI Overviews switch.
A rebuilt site was ready to launch with a robots.txt that blocked every crawler and without its llms.txt. We flagged it as a launch blocker with a go-live checklist, and the site went live with crawler access open.