They are not interchangeable, and blocking the wrong one removes you from answers you wanted to be in. Here is what each is for, in plain terms, and which decision each one actually affects.
- GPTBot — OpenAI. Crawls for model training and for retrieval. Blocking it is a training decision with a visibility cost.
- OAI-SearchBot — OpenAI. Builds the search index behind ChatGPT's answers. Blocking it removes you from those answers, and it does not train on what it fetches.
- ChatGPT-User — not a crawler. A live fetch made because a person in a conversation asked for that page.
- ClaudeBot — Anthropic's crawler.
- GoogleOther — Google's crawler for work that is not web search. This is the one most articles omit.
- Google-Extended — not a crawler at all. A robots.txt token controlling whether content Google already has may be used for Gemini.
OpenAI's three
OpenAI runs three separate agents and documents them separately, which matters because they do different jobs and take different robots.txt rules.
GPTBot is the general crawler. Content it collects may be used to improve future models. This is the one most blocking advice is about, and the one where a publisher's interests genuinely differ from ours.
OAI-SearchBot builds and maintains the index ChatGPT searches when it answers with current information. OpenAI is explicit that this one is not used for training. If you want ChatGPT to be able to find you and are not comfortable being training data, this is the distinction that lets you have one without the other.
ChatGPT-User shows up when someone in a conversation causes a fetch — a link pasted, or a browse triggered by the question. It is a user action, not a bulk crawl, and blocking it means a person who explicitly asked to look at your page cannot.
Anthropic and Google
ClaudeBot is Anthropic's web crawler and follows the usual robots.txt conventions.
Google is where the confusion lives. Googlebot is web search. GoogleOther is a separate crawler for everything else Google does, and if you are thinking about generative answers, this is the one to watch — it is a real crawler making real fetches, and it is missing from most published lists.
Google-Extended is different in kind. It fetches nothing. It is a token you put in robots.txt to control whether content Google has already crawled with Googlebot may be used for Gemini and related products. You will never see it in an access log, because it is a permission and not an agent — which is exactly why advice to “block Google-Extended if you don't want AI using your content” is confusing when people then go looking for it in their logs.
What we actually see
On our own site, over the last week, the crawlers that fetched a business page were ClaudeBot and GoogleOther. GPTBot and OAI-SearchBot fetched the home page and other public pages in the same period without reaching a business page — a discovery problem rather than a policy one, and a reminder that “which bots visit the site” and “which bots read the page you care about” are different questions with different answers.
The decision, put plainly
For a business that wants to be recommended, the calculus is not the same as for a publisher whose product is the writing itself.
- A publisher may reasonably block GPTBot: the content is the asset, and training on it is a transfer with nothing coming back.
- A local business is usually in the opposite position. Your hours and prices are not an asset to be protected — being quoted correctly is the entire point, and blocking a retrieval crawler removes you from the answer.
If you want the middle ground: allow OAI-SearchBot and block GPTBot. You stay findable in ChatGPT's answers and stay out of training. Whether that distinction holds over time is up to OpenAI, not to us, but it is the documented behaviour today.
Common questions
Should I block GPTBot?
If your business is the content — publishing, courses, research — there is a real argument for it. If you sell a service locally and want assistants to describe you correctly, blocking crawlers works against the thing you want. Consider allowing OAI-SearchBot even if you block GPTBot.
Why is Google-Extended never in my log?
Because it does not fetch anything. It is a robots.txt token that governs how content Googlebot already collected may be used, not a crawler that visits. Looking for it in an access log will always come up empty.
Does blocking a crawler remove me from answers already given?
Not retroactively in any way you control. Material already retrieved is already held, and taking a page down or disallowing a crawler afterwards does not withdraw it. This is the argument for deciding before publishing rather than after.
What does a permissive robots.txt look like?
Allow everything, and disallow only what should not be public — an admin area, a dashboard, an account page. That is what we run, and it is the right default for a business that wants to be found.
Are these the only AI crawlers?
No. PerplexityBot, Applebot-Extended, Bytespider, CCBot and others exist, and the list changes. The six here are the ones behind the assistants most people actually use, which is why they are the ones worth understanding first.
- OpenAI — Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic — Does Anthropic crawl data from the web
- Google Search Central — Google crawlers and fetchers
- Google Search Central — Google-Extended, on its role as a control rather than a crawler
Crawler behaviour and policy are set by the companies that operate them and change without notice; confirm against the documentation linked above before acting. The observations about which crawlers reached our pages are narubox measurements on one site. narubox is not affiliated with OpenAI, Anthropic or Google.