AI crawlers explained: OAI-SearchBot, GPTBot, Claude-SearchBot, PerplexityBot

AI crawlers do three different jobs. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index your pages so assistants can surface them. User triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User open a single page because someone asked. Training crawlers such as GPTBot and ClaudeBot collect content for model training. Blocking the wrong one removes you from AI answers.

Most robots.txt files were written before any of these existed, which is why sites end up invisible to one assistant and wide open to another. This page is the reference: who each bot is, what its documentation says it does, and what happens when you disallow it. For the strategy these choices serve, start with What is GEO? A practical guide.

Key takeaways

• Search crawlers, user triggered fetchers and training crawlers are three separate decisions, even when they belong to the same provider.
• A blanket Disallow: / for User-agent: * blocks the search crawlers that decide whether you appear in AI answers at all.
• Blocking training crawlers (GPTBot, ClaudeBot, the Google-Extended token) is a content licensing choice and does not remove you from AI search results.
• User agent strings can be faked, so verify traffic against the IP range files each provider publishes.

What are the three types of AI crawler?

The label "AI crawler" hides three activities with different consequences for your business.

1. Search crawlers build the index an assistant queries when it needs current information. Block them and you are not a candidate for that assistant's answers.
2. User triggered fetchers open one URL, in real time, because a person asked something that needs that page. No index, no schedule.
3. Training crawlers collect text that may train future models, with effects that show up over model generations rather than next week.

All three sit in the same robots.txt file and look interchangeable. They are not.

Which crawler does what?

Every token below comes from the provider's own documentation.

Token	Provider	Job it documents	What the docs say about robots.txt
OAI-SearchBot	OpenAI	Surfaces websites in ChatGPT search results	Sites opted out "will not be shown in ChatGPT search answers"
GPTBot	OpenAI	Crawls content to train generative AI foundation models	Disallowing it signals your content should not be used for training
ChatGPT-User	OpenAI	Handles user initiated actions in ChatGPT and custom GPTs	Because a user initiates them, "robots.txt rules may not apply"
OAI-AdsBot	OpenAI	Validates the safety of pages submitted as ads on ChatGPT	Only visits pages submitted as ads
Claude-SearchBot	Anthropic	Navigates the web to improve search result quality for Claude	Honors industry standard robots.txt directives
ClaudeBot	Anthropic	Collects web content that could contribute to model training	Honors robots.txt and the non standard Crawl-delay extension
Claude-User	Anthropic	Visits sites when a person asks Claude a question	Honors robots.txt directives
PerplexityBot	Perplexity	Surfaces and links websites in Perplexity results	Documented as the bot to allow; not used to crawl content for foundation models
Perplexity-User	Perplexity	Opens a page to answer a user and include a link	"Generally ignores robots.txt rules"
Googlebot	Google	Builds Google's search indexes	Standard robots.txt rules apply
Google-Extended	Google	robots.txt token for Gemini training and grounding	A control token with no separate user agent string

Search crawlers: the ones that gate your visibility

These are the bots to treat as you would treat Googlebot.

• OAI-SearchBot puts a site into ChatGPT's search results, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers (OpenAI). It is a separate token from GPTBot, so you can decline training and stay searchable.
• Claude-SearchBot navigates the web to improve the relevance and accuracy of search responses in Claude, and Anthropic documents that its bots honor standard robots.txt directives (Anthropic).
• PerplexityBot is designed to surface and link websites in Perplexity's results, and Perplexity states it is not used to crawl content for AI foundation models (Perplexity).
• Googlebot and Google-Extended are the Gemini case, and they work differently. Googlebot builds the Search index. Google-Extended is not a crawler: it has no user agent string of its own, and the token exists so publishers can manage whether crawled content may train future Gemini models or be used for grounding, meaning content served to the model at prompt time. Google is explicit that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google Search Central).

If those crawlers cannot reach your site, the rest of your work on AI visibility has nothing to act on. That is the first thing to rule out when an assistant never names you, as described in How to check if ChatGPT mentions your brand.

User triggered fetchers: ChatGPT-User, Claude-User, Perplexity-User

When a person asks an assistant about a specific page, a different agent goes and gets it. OpenAI documents ChatGPT-User for user initiated actions in ChatGPT and custom GPTs, Anthropic documents Claude-User for websites accessed while answering a user, and Perplexity documents Perplexity-User for visiting a page to give an accurate answer with a link.

Two practical points follow. First, robots.txt is a weaker instrument here: OpenAI writes that robots.txt rules may not apply to ChatGPT-User, and Perplexity writes that Perplexity-User generally ignores them, while Anthropic states its agents honor the directives. Second, this traffic is a demand signal rather than a crawl budget problem: one hit means a real person sent an assistant to your page.

Training crawlers: GPTBot, ClaudeBot and the Google-Extended token

GPTBot crawls content to train generative AI foundation models, ClaudeBot collects web content that could contribute to training Anthropic's models, and Google-Extended governs the training of future Gemini models. Perplexity documents no training crawler and states that neither of its agents collects content for foundation models.

Deciding here is a licensing question rather than an SEO one. Blocking costs you nothing in AI search visibility, because the search tokens are separate, but it does not erase what earlier models already learned and does not stop other sites from describing you, the slower half of the picture explained in How AI models choose which brands to cite. Publishers with licensing leverage often block. Companies that want to be recommended usually allow.

Does llms.txt tell AI crawlers what to read?

Not for Google Search. In its changelog of 15 June 2026, Google clarified that llms.txt files are not needed for Google Search, that they will not negatively or positively impact your visibility or rankings, and that keeping them for other services or systems that use them is fine (Google Search Central). The AI optimization guide says the same thing about new machine readable files, AI text files and special markup: Google Search does not use them, and there is no AI specific schema to add (Google Search Central).

OpenAI, Anthropic and Perplexity document robots.txt, not llms.txt, as the way to control their crawlers. The conclusion is unglamorous: robots.txt is the control surface that exists, and Google's own guidance is that SEO best practices stay relevant because its generative AI features are rooted in its core Search ranking and quality systems.

How to check what you are allowing today

1. Open your own robots.txt and read it against the table above. A single blanket Disallow: / under User-agent: * is the most common way sites remove themselves from AI search without meaning to.
2. Decide per job, not per provider. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. Choose separately on GPTBot, ClaudeBot and Google-Extended.
3. Confirm the bots can see content. A rule that allows a crawler is worthless if the page renders client side and returns an empty document, the failure mode documented in How AI models choose which brands to cite.
4. Verify the traffic in your logs. Each provider publishes the IP ranges its agents use, at openai.com/searchbot.json, openai.com/gptbot.json and www.perplexity.com/perplexitybot.json among others, because a user agent string alone proves nothing.
5. Re-measure afterwards. Access is the floor, not the result.

Once the right bots can read your pages, the open question is whether the assistants actually mention you. On paid plans, Huntair SEO tracks the prompts you choose across ChatGPT, Gemini, Claude and Perplexity together with the keywords you choose on Google SERP, so crawler access and its payoff sit in one place instead of two.

FAQ

Should I block GPTBot?

Only if you do not want your text used to train future models. It is a content licensing decision with no upside for visibility, and it does not affect ChatGPT search, because OpenAI documents OAI-SearchBot as the token that surfaces sites there.

Does blocking training crawlers hide me from AI search results?

No, as long as you keep the search crawlers allowed. OpenAI, Anthropic and Perplexity document separate tokens for search and for training, and Google states that Google-Extended does not affect inclusion or ranking in Google Search. The mistake to avoid is a blanket rule that catches every bot at once.

How do I tell a real AI crawler from a fake one?

By IP address, not by name. A user agent string is trivial to copy, so scrapers can announce themselves as GPTBot or PerplexityBot. OpenAI and Perplexity publish JSON files listing the IP ranges their agents use: match the request IP against them before you trust the label.

Do AI crawlers render JavaScript like Googlebot?

Generally no, which is why server side rendering or prerendering of public pages matters more for AI visibility than for classic search. The measurements behind that, and what they mean for a page that builds its content in the browser, are in How AI models choose which brands to cite.