AI Crawlers and robots.txt: Which Bots to Allow, GPTBot to Google-Extended

A robots.txt file is the one place where a software vendor decides, in writing, which AI systems may read its pages. The operators’ own documentation splits those systems into three jobs. Crawlers that collect training data, crawlers that build the search index an assistant queries, and fetchers that open a single page because a person asked.

Most advice treats all three as one “AI bot” and blocks or allows them together. This guide reads the documentation from OpenAI, Perplexity, Anthropic, Google, Common Crawl, Apple and Meta as it stood on 2026-09-04. It identifies each token and says what each is for.

It ends with the allow table and the file webpossible.com itself uses. If you market property-ops software, the practical output is a snippet and three questions to send your web team. The commercial version is the answer engine work for proptech.

Which AI crawlers should you allow in robots.txt?

Allow every crawler with a documented path to a recommendation. That means the search crawlers and the user-triggered fetchers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, Googlebot and Applebot. Decide the training bots on your training policy, because none of the documents read here says a Disallow on them changes what the search side shows.

This is the file webpossible.com serves. A wildcard allow, then each token named so the policy is explicit, then the sitemap.

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://webpossible.com/sitemap-index.xml

The live file at webpossible.com matches this: User-agent: * with Allow: /, an explicit Allow: / for the nine named tokens, and a pointer to the sitemap index. Nothing on the site is disallowed for anyone.

The file also says, in a comment, that a Markdown twin of every page sits at the page URL plus index.md. A curated map is at /llms.txt; the guide to llms.txt files explains both.

The allow table

Token	Operator	Job per its documentation	robots.txt per its documentation	webpossible.com
GPTBot	OpenAI	Training	Honored	Allow, named
OAI-SearchBot	OpenAI	ChatGPT search index	Honored, about 24 hours to adjust	Allow, named
ChatGPT-User	OpenAI	User-triggered fetch	May not apply	Allow, named
PerplexityBot	Perplexity	Perplexity search index	Allow recommended; no explicit statement	Allow, named
Perplexity-User	Perplexity	User-triggered fetch	Generally ignored	Allow, named
ClaudeBot	Anthropic	Training	Honored	Allow, named
Claude-User	Anthropic	User-triggered fetch	Honored	Allow, named
Claude-SearchBot	Anthropic	Search quality	Honored	Allow, wildcard
Google-Extended	Google	Training and grounding control token	Token only, no crawler	Allow, named
Googlebot	Google	Search index, AI Overviews, AI Mode	Honored	Allow, wildcard
CCBot	Common Crawl	Open crawl archive	Disallow given as the opt-out	Allow, named
Applebot-Extended	Apple	Training control token	Token only, no crawler	Allow, wildcard
Meta-ExternalAgent	Meta	Training and indexing	Subject to rules	Allow, wildcard
Meta-ExternalFetcher	Meta	User-triggered fetch	May bypass	Allow, wildcard

Our own position is to allow every one of them. A training opt-out protects text a company wants to sell, and a vendor page exists to be read.

What are the three kinds of AI bot?

Each operator sorts its bots by job, and the job matters more than the operator. A training crawler collects pages that may end up in a model’s training data, and blocking it is an opt-out from training. A search crawler builds the index the assistant consults when it looks something up, and blocking it removes your pages from that index.

A user-triggered fetcher opens one URL because a person asked a question that needed it, or pasted the link. Several operators say this third kind may not honor robots.txt at all, because the request came from a person.

For a vendor that wants to be named in an answer, the second and third kinds carry the recommendation. The first kind has no documented path to a citation. Each section below checks that against a specific document.

What are GPTBot, OAI-SearchBot and ChatGPT-User?

OpenAI documents three tokens on its bots page (developers.openai.com/api/docs/bots). It is the clearest of the seven operators about which does what.

GPTBot

Per OpenAI’s bots documentation, GPTBot identifies with a user-agent string ending in GPTBot/1.4; +https://openai.com/gptbot and is “used to make our generative AI foundation models more useful and safe.” Disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” That is the training crawler.

OAI-SearchBot

The same OpenAI page says OAI-SearchBot, whose string ends in OAI-SearchBot/1.4; +https://openai.com/searchbot, is “used to surface websites in search results in ChatGPT’s search features.” It adds that “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.

That is the search crawler. Its sentence about opted-out sites is the most direct statement any operator makes about what a Disallow costs.

ChatGPT-User

ChatGPT-User, per the same documentation, carries a string ending in ChatGPT-User/1.0; +https://openai.com/bot and is used “for certain user actions in ChatGPT and Custom GPTs.” OpenAI says that “because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI also states that ChatGPT-User “is not used to determine whether content may appear in Search.”

So a Disallow for ChatGPT-User does not opt you out of training and does not remove you from search. It is a signal to a fetcher that may not read it.

OpenAI’s bots page publishes the IP ranges for each bot as JSON at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Those files are what you check a log line against, covered below. Which of the consulted pages become citations is in the guide to ranking in ChatGPT.

What are PerplexityBot and Perplexity-User?

Perplexity documents two bots (docs.perplexity.ai/guides/bots), and neither is described as a training crawler.

Per Perplexity’s crawler documentation, PerplexityBot identifies as PerplexityBot/1.0; +https://perplexity.ai/perplexitybot. It “is designed to surface and link websites in search results on Perplexity” and “is not used to crawl content for AI foundation models.” The same page recommends allowing PerplexityBot in robots.txt and publishes its IP ranges at perplexity.com/perplexitybot.json.

The page contains no sentence saying PerplexityBot honors robots.txt. It tells you to allow it, which implies the file is read, and we leave it there.

Perplexity-User, per the same documentation, identifies as Perplexity-User/1.0; +https://perplexity.ai/perplexity-user and may visit a page when a user’s question needs it. It “is not used for web crawling or to collect content for training AI foundation models,” and “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity lists its IP ranges at perplexity.com/perplexity-user.json.

That robots sentence is unusual for its bluntness. Stopping those fetches takes a server-side rule keyed to the user-agent or the IP list. A vendor that writes one has chosen to fail the request of a person who asked about its product.

How those fetches become numbered citations is in the Perplexity citation guide.

What are ClaudeBot, Claude-User and Claude-SearchBot?

Anthropic documents three bots in a help center article (support.claude.com, article 8896518, “Does Anthropic crawl data from the web”).

Per that article, ClaudeBot collects “web content that could potentially contribute to” training. Claude-User is the agent Claude may use to access websites “when individuals ask questions to Claude,” and Claude-SearchBot “navigates the web to improve search result quality for users.” Training, fetch, search, in that order.

The article says Anthropic’s bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” and gives the User-agent: ClaudeBot / Disallow: / form for each bot separately. It warns that blocking by IP address “may not work correctly or persistently guarantee an opt-out.” Anthropic publishes its source IPs at claude.com/crawling/bots.json.

Two things the article does not do. It does not print full user-agent strings, only the three token names, so a log filter should match on the token. It does not say Claude-User ignores robots.txt, which puts Anthropic’s fetcher on the opposite side from OpenAI’s and Perplexity’s.

Anthropic’s web fetch tool documentation also names robots.txt as a reason a fetch can be refused. The guide to how Claude picks vendor sources covers that tool.

How does Google-Extended work?

Google is the operator most often misread, because its AI control is not a bot.

Per Google’s crawler documentation (“Google’s common crawlers”), Google-Extended is a standalone product token for robots.txt that “doesn’t have a separate HTTP request user agent string.” Crawling “is done with existing Google user agent strings.”

The same page says the token controls whether content trains future Gemini models and is used for grounding. It “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

You will never see Google-Extended in a log, only Googlebot. Google’s AI features documentation (“AI features and your website”) says AI is built into Search. Robots.txt directives for Googlebot are therefore “the control for site owners to manage access to how their sites are crawled for Search.”

The same Google page points anyone who wants less of a page shown to the “nosnippet, data-nosnippet, max-snippet, or noindex controls.” Per Google’s crawler page, preferences addressed to Googlebot “affect Google Search (including Discover and all Google Search features).”

Read together, the two pages mean a vendor cannot opt out of AI Overviews or AI Mode while staying in the ten blue links. The documented control is the same crawler and the same snippet directives. What that means for the answer box is in the AI Overviews guide.

What about CCBot, Applebot and Meta’s crawlers?

Common Crawl’s CCBot page (commoncrawl.org/ccbot) gives the user-agent CCBot/2.0 (https://commoncrawl.org/faq/), describes “an open repository of web crawl data,” and gives User-agent: CCBot / Disallow: / as the way to prevent crawling. CCBot is not an answer engine. Its archive is a public dataset, which is why it belongs with the training crawlers.

Apple’s Applebot page (support.apple.com/en-us/119829) says Applebot’s data powers “Spotlight, Siri, and Safari” and that Applebot “respects standard robots.txt directives.” Applebot-Extended is “a secondary user agent” that lets publishers opt out of training generative models.

The same page says “webpages that disallow Applebot-Extended can still be included in search results.” The shape matches Google’s: one crawler and one control token for training.

Meta’s web crawlers page lists meta-externalagent/1.1, which crawls “for use cases such as training foundation AI models or improving products by indexing content directly.” It also lists meta-externalfetcher/1.1, which “fetches individual links at a user’s request” and “may bypass robots.txt rules.” Per that page, Meta’s training crawl and its indexing crawl share one token, so a vendor cannot separate the two the way it can with OpenAI’s three.

Does blocking a training bot remove you from search-grounded answers?

Not per any document read here. The operators separate the controls, and the separation is the whole basis for the claim. OpenAI’s bots documentation ties the training opt-out to GPTBot and appearance in ChatGPT search answers to OAI-SearchBot, and says nothing that links the two.

Google’s crawler documentation says Google-Extended does not affect inclusion in Google Search, and its AI features page says Googlebot’s directives are the control for Search, AI features included. Apple’s page says pages that disallow Applebot-Extended can still appear in search results. Anthropic’s article gives a separate Disallow block for each of its three bots and describes the three jobs separately.

The documents stop there. None promises that a page blocked from training will be treated identically at answer time, and none describes what the model already knows from earlier training.

Our hypothesis is that a vendor whose pages were in a model’s training data can be named from memory with no search at all. A vendor who blocks training crawlers today gives up that path for future models while keeping the search-grounded one. We have not tested it and will not claim it until we have run prompts with search disabled.

What the Index can see

Search-grounded is the path the Index measures. In a run of one maintenance-software prompt captured through the OpenAI API on 3 September 2026, ChatGPT consulted 124 URLs, 38 of them on reddit.com. That run is retired from the edition and kept in the archive.

It is cited here because every one of those 124 pages had to be fetchable to be consulted. A robots rule or a firewall rule that stopped the fetch would have removed the page before the model read anything. How the Index reads consulted lists and citations is on how we measure AI recommendations.

Do user-triggered fetchers honor robots.txt?

Three operators’ documentation says their fetcher may not, and Anthropic’s says its fetcher does. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores robots.txt rules,” and Meta says Meta-ExternalFetcher “may bypass robots.txt rules.” Anthropic’s article says its bots, Claude-User among them, honor robots.txt directives.

The practical reading is that robots.txt is the wrong tool for controlling fetchers and the right tool for controlling crawlers. A fetch happens because a property manager asked an assistant about maintenance software, or pasted your pricing URL. Allowing it costs one page load.

Blocking it at the server means that person’s assistant reports that your page could not be read. The assistant then answers from whatever else it found, which in the run above was mostly a forum.

How do you verify in server logs what reached you?

Documentation says what a bot is supposed to do. Logs say what reached you. There are four checks, in order.

1. Filter on the token names: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot, Googlebot, Applebot, CCBot, meta-externalagent, meta-externalfetcher. Match on the token, not the full string, because version numbers and the browser prefix change.
2. Check the source IP against the published lists. OpenAI, Perplexity and Anthropic each publish a JSON file, given in their sections above. A line carrying the GPTBot token from an address outside openai.com/gptbot.json is someone else wearing the name.
3. Read the status codes per bot. A 200 is a page read. A 403 is a refusal. Most 403s to AI bots are not written in robots.txt; they come from a CDN bot-management setting, a firewall rule, or a rate limiter that matched the user-agent. A 301 chain or a 5xx on the pricing page deserves the same attention.
4. Count fetches by path. If Perplexity-User and ChatGPT-User are opening your pricing and integrations pages, people are asking assistants about you and the assistants are looking. If the fetchers only hit the homepage, the questions do not reach the pages with the facts, which is a content problem covered in the AI visibility guide.

What a 403 costs

What a 403 to a search bot costs is stated most directly by OpenAI. Per OpenAI’s bots documentation, sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. A firewall 403 is not the same signal as a robots Disallow, and the OpenAI sentence is about the Disallow, so we do not claim the identical outcome.

What a 403 does, without dispute, is deny the page to the crawler on that request, and a page that cannot be read cannot be indexed from that read. For a fetcher, the cost is the single answer in front of the single person who asked. The audit of what AI says about a vendor starts by reading these same log lines.

What should a marketing lead send the web team?

Most vendor marketing leads do not edit robots.txt or read access logs. Three questions and the snippet above are enough to get the right answer back.

• Does our robots.txt allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and Claude-User, and is anything under /pricing or /integrations disallowed?
• In the last 30 days of access logs, what status codes did those tokens receive, and did any CDN or firewall rule return a 403 to them?
• Which pages did ChatGPT-User and Perplexity-User fetch most?

The third answer is the useful one. It tells you which of your pages assistants open when a buyer asks about you, which is the list of pages that need the facts on them.

Limits

This page is documentation, read on one day. Every quoted sentence comes from a page named in the sentence, and every one of those pages can change without notice. Nothing here reports what any bot did on any particular site beyond the one archived Index run cited above.

The documents describe intent, not enforcement. A statement that a bot honors robots.txt is the operator’s statement. Only your logs can confirm it for your domain.

Robots.txt is advisory. It is read by bots that choose to read it, and the operators above say their fetchers may not. A vendor that needs a hard block needs a server rule, and a vendor that wants to be recommended should not need one.

The table reflects webpossible.com’s file, which is a publisher’s choice. A vendor with a different view of training can disallow GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent and, per the documents above, keep every search and fetch path open. Blocking OAI-SearchBot, PerplexityBot or Googlebot is a different decision with a documented cost.

Structured data and the Markdown twin do not change any of this. A page must be reachable before markup on it matters; the structured data guide assumes the fetch succeeded.

What changes month to month?

Tokens and versions. OpenAI’s bots page carried 1.4 for GPTBot and OAI-SearchBot on the day of reading, and the browser prefix on OAI-SearchBot names a Chrome version. Match on tokens in any filter you keep.

Robots policy per fetcher. The three operators whose fetchers may ignore robots.txt could change that position, and Anthropic could change its own. Re-read the four documents before repeating the table.

The IP lists. The JSON files are the source of truth for whether a request is genuine, and they rotate. A verification script should fetch the list each run rather than cache it.

Google’s AI features. Google’s AI features documentation says Googlebot’s robots directives and the snippet controls are the controls for AI in Search. If Google ever introduces a separate token for those features, this page’s Google section is wrong from that day.

The documentation URLs. OpenAI’s bots page moved from platform.openai.com to developers.openai.com and Anthropic’s help center moved from support.anthropic.com to support.claude.com before 2026-09-04. Both old URLs redirect; cite the current ones.

Related guides

The engine pages read the answer side of what these bots feed. The guide to ranking in ChatGPT covers what the links in an answer leave out, and the Perplexity citation guide covers dated results and numbered citations. The AI Overviews guide covers the answer box Googlebot governs, and the guide to how Claude picks vendor sources covers Anthropic’s fetch tool.

The agent legibility guide covers the browser agents that act for a buyer and mostly skip robots.txt.

The guide to llms.txt files covers the map file this site’s robots.txt comments point to. The structured data guide covers what to put on the page once the fetch succeeds. Metric definitions live in the AI visibility guide, and the pillar on getting named by AI assistants puts the pieces in order.

Changelog

• 2026-10-28: Reformatted with a short answer, the allow-all file printed near the top, and question headings with one subsection per OpenAI bot. Added the four log checks as a list, a section for marketing leads, and an FAQ. Documentation quotes unchanged from the 2026-09-04 reading.
• 2026-09-26: The September figures now cite the 25 ChatGPT app captures per category; the API run of 3 September 2026 is described as retired from the edition and archived.
• 2026-09-04: First published.