Which AI crawlers to allow in robots.txt (and which ones only train models)

To stay visible in AI answers, allow the crawlers that feed search and answers: OAI-SearchBot (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews, AI Mode), Bingbot (Bing and Copilot) and Applebot (Siri, Spotlight, Safari). GPTBot, ClaudeBot, Google-Extended and Applebot-Extended control training only, according to each company's documentation. You can block those without losing search visibility, though for Google-Extended there is a catch.

Three kinds of AI crawlers

AI companies now run several bots each, and they do different jobs. Treating them as one group ("block AI") can remove a site from AI answers when the owner only meant to opt out of training. The documentation from OpenAI, Anthropic, Perplexity, Google and Apple splits them into three kinds:

• Search and answer crawlers build the index an assistant searches when it answers a question. Block these and the assistant cannot cite your pages.
• Training crawlers or tokens decide whether your content may be used to train future models. Blocking them does not remove you from search, according to the vendors.
• User-triggered fetchers open a specific page because a person asked the assistant to, for example by pasting a link. Some of them do not follow robots.txt, because the visit is treated as a user action.

Per-bot reference table

Based on each company's documentation as read on November 9, 2026. Vendors add and rename bots, so check the linked pages before you change a live file.

OpenAI: OAI-SearchBot is the one that matters for ChatGPT search

OpenAI's crawler documentation is explicit. OAI-SearchBot shows websites in ChatGPT's search features, and sites that opt out of it will not be shown in ChatGPT search answers. GPTBot crawls content that may be used to train OpenAI's foundation models; disallowing it signals that the site should not be used in training. OpenAI states that each setting is independent of the others, so a site can allow OAI-SearchBot and disallow GPTBot. If both are allowed, OpenAI may use one crawl for both purposes. OpenAI also notes that search changes can take about 24 hours to take effect after a robots.txt update.

ChatGPT-User is different. It handles visits started by a user, and OpenAI says robots.txt rules may not apply to it.

The Bing index matters for ChatGPT too. BrightLocal's study of ChatGPT local answers found that ChatGPT search relied mostly on the Bing index (December 2024). A site that blocks Bingbot gives up that route as well.

Anthropic: three bots, all following robots.txt

Anthropic's help page lists ClaudeBot for training, Claude-SearchBot for indexing sites to improve search results, and Claude-User for fetching pages when a user asks a question. Anthropic says its bots honor robots.txt directives. Blocking ClaudeBot signals that the site's future content should be excluded from training datasets. Blocking Claude-SearchBot means content will not be indexed for search, and blocking Claude-User means pages will not be retrieved for user queries; Anthropic says both may reduce visibility in Claude's answers.

Two practical notes from the same page. Anthropic supports the non-standard Crawl-delay directive, with ClaudeBot as its example. And it advises against blocking Anthropic's IP addresses, because that can stop the bots from reading robots.txt at all. Each subdomain needs its own robots.txt.

Perplexity: allow PerplexityBot

Perplexity's crawler documentation says PerplexityBot is designed to show and link websites in Perplexity search results and is not used to crawl content for AI foundation models. Perplexity recommends allowing it for sites that want to appear in results. Perplexity-User visits pages when a user asks a question, and the documentation says it generally ignores robots.txt rules. It is also not used for training.

For local businesses, Perplexity also reads licensed data. It has shown Yelp data since March 2024 and licenses Tripadvisor reviews and listings since January 2025. Robots.txt only covers your own website, not those listings.

Google: Googlebot for search, Google-Extended for Gemini

Google has no separate crawler for AI. AI Overviews and AI Mode use the regular search index. Google's guidance on AI features says there are no additional requirements to appear in them beyond being indexed and eligible for a snippet. To limit what appears, Google points to nosnippet, data-nosnippet, max-snippet and noindex, not to a robots.txt token.

Google-Extended, described on Google's common crawlers page, is a robots.txt token with no separate user agent. It controls whether content may be used to train future Gemini models, and it also governs grounding: content from the Search index given to the model at prompt time in Gemini Apps and in Grounding with Google Search on Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal.

That second use is the catch for a business owner. Disallowing Google-Extended is not only a training opt-out. It also limits the use of your pages to ground answers in the Gemini app.

Bingbot and Applebot

Microsoft describes the Bing Places listing as the business listing for Bing search and Maps (October 2025); Copilot searches with Bing. Bingbot builds the Bing index. Bing's webmaster blog notes that Bingbot reads robots.txt at the root of each host, and that a group written specifically for bingbot makes it ignore the other directives in the file.

Apple's Applebot page says Applebot data powers search in Spotlight, Siri and Safari. Applebot-Extended does not crawl pages; it lets publishers opt out of training Apple's generative models, and Apple says pages that disallow it can still be included in search results.

A recommended robots.txt

The file below allows every search and answer crawler from the table, and shows the training-only opt-outs as an optional block. Replace the example paths and domain with your own.

One rule to know first. Under the robots.txt standard, RFC 9309, a crawler that finds a group naming it follows that group and ignores the User-agent: * group. So any path you want to keep private has to be repeated in each named group.

# Search and answer crawlers: allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /cart/

# Optional: opt out of model training only.
# Delete this block if you are fine with training use.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
Disallow: /

# Google-Extended also limits grounding in Gemini Apps.
# Uncomment only if you accept that.
# User-agent: Google-Extended
# Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Listing several User-agent lines above one set of rules is allowed by RFC 9309 and keeps the file short. After you publish it, open yoursite.com/robots.txt in a browser to confirm it is live, and remember that OpenAI says search changes take about a day.

Blocking training crawlers is a choice, not a fix

There is no single right answer for training bots. Reasons to block them: you do not want your text or photos used to train commercial models, or you want to reduce crawl load. Reasons to allow them: a model that has read your site may describe your business more accurately when it answers without searching, and for Google-Extended, blocking also limits grounding in the Gemini app. Either choice is legitimate. What does not make sense is blocking the search crawlers by accident while trying to block training.

When we audit a site at AIReputa, robots.txt is one of the first files we read, because a single line can make a business invisible to an assistant regardless of its reviews or listings.