Free tool · no sign-up

Free AI crawler check: can ChatGPT, Claude and Perplexity read your site?

Enter your domain to see whether ChatGPT, Claude, Perplexity and 10 other AI and search crawlers may read your site. The check reads your robots.txt and llms.txt, loads your home page as GPTBot, ClaudeBot and PerplexityBot, and tells you in plain words what to fix.

We store the domain you check and the time, not your IP address.

Updated

How it works

What does the free AI crawler check do?

It reads the public files that decide whether AI crawlers may visit your site, then tests your home page the way three of those crawlers would ask for it. Four steps, in up to 30 seconds:

  1. robots.txt. It reads /robots.txt and works out, for each of the 13 crawlers below, whether the rules allow it to read your site root.
  2. llms.txt. It looks for /llms.txt, a short Markdown guide to your site for AI tools.
  3. Your home page. It loads the home page like a browser and notes the HTTP status, a noindex rule (meta robots tag or X-Robots-Tag header), JSON-LD structured data, and how much text is there before any JavaScript runs.
  4. Live test. It loads the home page again with the published user agents of GPTBot, ClaudeBot and PerplexityBot and compares the answers. A firewall or CDN rule that turns a crawler away shows up here even when robots.txt allows it.

Limits

What the check cannot tell you

  • It reads only public files and your home page. Other pages are not checked.
  • robots.txt rules are interpreted for the site root. A rule that blocks only a folder, such as /blog/, is not reported.
  • The live test runs from our server with each crawler’s published user agent. A firewall that checks IP addresses may treat our server differently from the real crawlers, in either direction.
  • It shows whether crawlers may read your site, not whether AI assistants mention you. That depends on your content, your sources elsewhere and your competitors.
  • Results are a snapshot. A repeat check of the same domain within 10 minutes shows the same result.

We store the domain you check and the time, not your IP address.

The crawlers

What are AI crawlers, and which 13 does the check cover?

AI crawlers are bots that fetch web pages for AI companies. They do three different jobs: training crawlers collect pages to train models, search crawlers build the index an assistant searches when it answers with sources, and user agents fetch a page when a person asks an assistant about it. Each has its own name in robots.txt, so a site can allow one and block another.

CrawlerOwnerWhat it is forIn this check
GPTBotOpenAITraining OpenAI’s modelsrobots.txt and live test
OAI-SearchBotOpenAIChatGPT search: pages ChatGPT can show and citerobots.txt
ChatGPT-UserOpenAIVisits a page when a ChatGPT user asks for itrobots.txt
ClaudeBotAnthropicTraining Anthropic’s modelsrobots.txt and live test
Claude-SearchBotAnthropicClaude searchrobots.txt
Claude-UserAnthropicVisits a page when a Claude user asks for itrobots.txt
PerplexityBotPerplexityPerplexity searchrobots.txt and live test
Perplexity-UserPerplexityVisits a page when a Perplexity user asks for itrobots.txt
Google-ExtendedGoogleGemini training and groundingrobots.txt
GooglebotGoogleGoogle Search and AI Overviewsrobots.txt
BingbotMicrosoftBing search and Copilotrobots.txt
Applebot-ExtendedAppleApple Intelligence trainingrobots.txt
CCBotCommon CrawlAn open web dataset used by many modelsrobots.txt

The three crawlers marked “live test” are also tested with a real request. For the other ten the check reads robots.txt only.

FAQ

Questions about the AI crawler check

Is the AI crawler check free?

Yes. The check is free and needs no sign-up. It reads your robots.txt, your llms.txt and your home page, and loads the home page as GPTBot, ClaudeBot and PerplexityBot. We store the domain you check and the time, not your IP address. To keep the service fair, the number of checks per visitor in a few minutes is limited.

Should I block AI crawlers?

It is a business decision, and the crawlers do different jobs. Blocking a training crawler such as GPTBot or ClaudeBot keeps your pages out of that company’s future training data. Blocking a search or user crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can keep your pages out of that assistant’s answers and citations. If you want to be cited, allow the search crawlers.

robots.txt allows a crawler, but the live test says it is refused. Why?

A firewall, CDN or security plugin can refuse a request by its user agent even when robots.txt allows it. If our normal request gets your page and the same request with a crawler’s user agent gets an error, a rule on your server or CDN is the likely cause. Our server is not the crawler’s own address, so check the rule itself before you change anything.

Does passing the check mean AI assistants will mention my company?

No. Access is the first step: a crawler that cannot read your site cannot quote it. Whether an assistant names you depends on your content, your sources elsewhere and your competitors. The AI Search Visibility Audit measures that: 20 buyer questions on ChatGPT, Perplexity, Claude and Grok, 3 runs each, every answer dated, against up to 3 competitors.

What is llms.txt, and do I need one?

llms.txt is a proposed file at the root of a site that gives AI tools a short Markdown overview of the site with links to its key pages (llmstxt.org). Not every assistant reads it, so treat it as a cheap extra, not a requirement. Crawler access and pages that show their text without JavaScript come first.

What does the free check not cover?

It checks your home page and the site-wide files only. It does not read your other pages, review structured data in depth, judge your content or ask assistants about you. The AI Search Readiness Audit at $149 (€129) checks crawler access plus content, structured data and citability on your key pages, with a ranked fix plan, in 3 business days.

Crawlers can read your site. Do assistants recommend it?

The Readiness Audit checks crawler access plus content, structured data and citability, for $149€129. The Visibility Audit adds 240 dated answers from ChatGPT, Perplexity, Claude and Grok and you against up to 3 competitors, for $290€249.

Businesses only. Prices exclude VAT. Not ready to buy? The sample report is a real audit of another company, anonymised.

Get your audit