AI crawlers can read your site only if your robots.txt file allows them. Each AI company uses its own crawler names, and blocking the wrong one can remove you from AI search results, while blocking another only stops your pages being used for model training. This guide lists the main crawlers, what each controls and how to check your own file in a few minutes.
A page an engine cannot fetch is a page it is much less likely to cite, so this check belongs at the start of any AEO work.
What is robots.txt and what can it do?
robots.txt is a plain text file at the top level of your site, at /robots.txt, that tells crawlers which paths they may fetch. The standard behind it is RFC 9309, which also makes a point worth remembering: the rules are not a form of access authorization (RFC 9309). They ask well-behaved crawlers to stay out. They do not lock anything. Pages that must stay private need real protection, such as a login.
Which AI crawlers matter?
Different crawlers do different jobs, and the names are easy to confuse. Here are the main ones, as documented by their owners:
| User agent | Owner | What it does | Source |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | OpenAI |
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | OpenAI |
ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT something that needs it | OpenAI |
Googlebot | Crawling for Google Search, including its AI features | ||
Google-Extended | A token controlling whether crawled content is used for future Gemini training and grounding | ||
ClaudeBot | Anthropic | Collects public content that may contribute to training Anthropic's models | Anthropic |
Claude-User | Anthropic | Retrieves pages when a person asks Claude something that needs them | Anthropic |
Claude-SearchBot | Anthropic | Crawls the web to improve the relevance of search results | Anthropic |
A few points from those sources:
- OpenAI's settings are independent. A site can allow
OAI-SearchBotto appear in ChatGPT search results while disallowingGPTBot. OpenAI also notes that changes for search results can take about 24 hours to take effect, and that robots.txt rules may not apply toChatGPT-User, which acts on a person's request. - Google separates training from Search. Google says
Google-Extendeddoes not affect a site's inclusion in Google Search and is not a ranking signal. Crawling preferences forGooglebotaffect Google Search. - Google's AI Overviews and AI Mode come through Search. Google says a page needs to be indexed and eligible to appear in Search with a snippet to appear in them, with no additional requirements (Google Search Central). Blocking
Googlebotis therefore a far bigger step than blockingGoogle-Extended. - Anthropic splits training from search and user requests. It notes that disabling
Claude-SearchBotorClaude-Usermay reduce your visibility in those results.
How do you check your robots.txt?
- Open the file. Visit
yourdomain.com/robots.txtin a browser. - Look for blanket blocks. A rule such as
User-agent: *followed byDisallow: /blocks every compliant crawler from the whole site. - Look for named groups. Search for the user agents above. A group naming one of them can override the general rules.
- Check subdomains. Each subdomain has its own robots.txt. A blog or docs site on a subdomain needs its own check.
- Check that the file loads. Under RFC 9309, if a robots.txt file cannot be reached because of a server error, crawlers must assume the whole site is disallowed. If the file is simply missing (a 4xx response), crawlers may access anything (RFC 9309). A broken file is worse than none.
What does a sensible robots.txt look like?
The right choice depends on your goals. This example is only an illustration of one common setup. It allows ChatGPT's search crawler, and asks OpenAI's training crawler to stay out:
# Example only: choose what fits your business
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
If your priority is being named in AI answers, avoid blocking the crawlers that power search and retrieval. Decide separately whether you want your content used for model training, because the two choices are independent for OpenAI and Google.
What else can stop an AI engine from reading a page?
robots.txt is only one barrier. Pages can also be unreachable because of:
- A login or paywall in front of the content
noindexor snippet limits, which Google says apply to how content appears in its AI features- Redirect loops or server errors
Google also notes that more restrictive permissions limit how your content is featured in its AI experiences (Google Search Central blog, May 2025).
How does Verte check crawler access?
The Verte site audit checks what AI crawlers can reach and read on your site and flags what needs fixing, such as a crawler blocked in robots.txt. It sits beside your daily AI visibility tracking, so you can see whether an access problem lines up with prompts where you are missing.
Access is only one input. Read AI citation gaps to see what the engines cite once they can read you, and Do you need an llms.txt file? for the file everyone asks about. The difference between Google's two AI features is in Google AI Overviews vs AI Mode, and AEO vs SEO covers how crawling fits with the rest of your search work.
Start tracking your business or start as an agency. See current plans.