AI Crawlers & robots.txt: GPTBot, PerplexityBot, ClaudeBot, Google-Extended

A robots.txt file is the one place where a software vendor decides, in writing, which AI systems may read its pages. The decision has three parts, and the operators’ own documentation keeps them separate: crawlers that collect content for model training, crawlers that build the search index an assistant queries at answer time, and fetchers that open a single page because a person asked a question. Most advice we see treats all three as one “AI bot” and blocks or allows them together. This guide reads the documentation from OpenAI, Perplexity, Anthropic, Google, Common Crawl, Apple, and Meta as it stood on 2026-09-04, names each token, says what each is for, and ends with the allow table webpossible.com itself uses. The mechanism these bots feed is on the pillar on getting named by AI assistants. The commercial version for property software vendors is the answer engine work for proptech.

Three jobs, three kinds of bot

Each operator sorts its bots by job, and the job matters more than the operator. A training crawler collects pages that may end up in a model’s training data, and blocking it is an opt-out from training. A search crawler builds the index the assistant consults when it decides to look something up, and blocking it removes your pages from that index. A user-triggered fetcher opens one URL because a person asked a question that needed it, or pasted the link. Several operators say this third kind may not honor robots.txt at all, because the request came from a person.

For a vendor that wants to be named in an answer, the second and third kinds carry the recommendation. The first kind has no documented path to a citation. That is the argument of this page, and each section below checks it against a specific document.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI documents three tokens on its bots page (developers.openai.com/api/docs/bots), and it is the clearest of the seven operators about which does what.

Per OpenAI’s bots documentation, GPTBot identifies with a user-agent string ending in GPTBot/1.4; +https://openai.com/gptbot, is “used to make our generative AI foundation models more useful and safe,” and disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” That is the training crawler.

The same OpenAI page says OAI-SearchBot, whose string ends in OAI-SearchBot/1.4; +https://openai.com/searchbot, is “used to surface websites in search results in ChatGPT’s search features,” and that “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” OpenAI adds that it can take about 24 hours from a robots.txt update for its systems to adjust. That is the search crawler, and its sentence about opted-out sites is the most direct statement any operator makes about what a Disallow costs.

ChatGPT-User, per the same OpenAI documentation, carries a string ending in ChatGPT-User/1.0; +https://openai.com/bot, is used “for certain user actions in ChatGPT and Custom GPTs,” and “because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI also states that ChatGPT-User “is not used to determine whether content may appear in Search.” So a Disallow for ChatGPT-User does not opt you out of training and does not remove you from search. It is a signal to a fetcher that may not read it.

OpenAI’s bots page publishes the IP ranges for each bot as JSON at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json. Those files are what you check a log line against, covered below. What happens after the crawl, which of the consulted pages become citations, is in the ChatGPT source-selection guide.

Perplexity: PerplexityBot and Perplexity-User

Perplexity documents two bots (docs.perplexity.ai/guides/bots), and neither is described as a training crawler.

Per Perplexity’s crawler documentation, PerplexityBot identifies as PerplexityBot/1.0; +https://perplexity.ai/perplexitybot, “is designed to surface and link websites in search results on Perplexity,” and “is not used to crawl content for AI foundation models.” The same Perplexity page recommends allowing PerplexityBot in robots.txt and publishes its IP ranges at perplexity.com/perplexitybot.json. The page contains no sentence saying PerplexityBot honors robots.txt. It tells you to allow it, which implies the file is read, and we leave it there.

Perplexity-User, per the same documentation, identifies as Perplexity-User/1.0; +https://perplexity.ai/perplexity-user, may visit a page when a user’s question needs it, “is not used for web crawling or to collect content for training AI foundation models,” and “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity lists its IP ranges at perplexity.com/perplexity-user.json.

That robots sentence is unusual for its bluntness. A Disallow line for Perplexity-User is, per Perplexity’s documentation, generally ignored. Stopping those fetches takes a server-side rule keyed to the user-agent or the IP list, and a vendor that writes one has chosen to fail the request of a person who asked about its product. How those fetches become numbered citations is in the Perplexity citation guide.

Anthropic: ClaudeBot, Claude-User, Claude-SearchBot

Anthropic documents three bots in a help center article (support.claude.com, article 8896518, “Does Anthropic crawl data from the web”).

Per that Anthropic article, ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training,” Claude-User is the agent Claude may use to access websites “when individuals ask questions to Claude,” and Claude-SearchBot “navigates the web to improve search result quality for users.” Training, fetch, search, in that order.

The Anthropic article says “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” gives the User-agent: ClaudeBot / Disallow: / form for each bot separately, and warns that blocking by IP address “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” Anthropic publishes its source IPs at claude.com/crawling/bots.json.

Two things the article does not do. It does not print full user-agent strings, only the three token names, so a log filter should match on the token. And it does not say Claude-User ignores robots.txt; it says the bots honor it, which puts Anthropic’s fetcher on the opposite side from OpenAI’s and Perplexity’s on that point. Anthropic’s web fetch tool documentation also names robots.txt as a reason a fetch can be refused, which the guide to how Claude picks vendor sources covers.

Google: Google-Extended is a token, Googlebot is the crawler

Google is the operator most often misread, because its AI control is not a bot.

Per Google’s crawler documentation (developers.google.com, “Google’s common crawlers”), Google-Extended is a standalone product token for robots.txt, “doesn’t have a separate HTTP request user agent string,” and crawling “is done with existing Google user agent strings.” The same Google page says the token controls whether content may be used “for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding,” and that “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

You will never see Google-Extended in a log. You will see Googlebot, and per Google’s AI features documentation, Googlebot’s robots directives govern the AI features inside Search. Google’s AI features documentation (developers.google.com, “AI features and your website”) says “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search,” and points anyone who wants less of a page shown to “nosnippet, data-nosnippet, max-snippet, or noindex controls.” Per Google’s crawler page, crawling preferences addressed to Googlebot “affect Google Search (including Discover and all Google Search features).”

Read together, Google’s two pages mean a vendor cannot opt out of AI Overviews or AI Mode while staying in the ten blue links; the documented control is the same crawler and the same snippet directives. What that means for the answer box is in the AI Overviews guide.

Common Crawl, Apple, Meta

Common Crawl’s CCBot page (commoncrawl.org/ccbot) gives the user-agent CCBot/2.0 (https://commoncrawl.org/faq/), describes the operator as a non-profit that maintains “an open repository of web crawl data,” and gives User-agent: CCBot / Disallow: / as the way to prevent crawling. CCBot is not an answer engine. Its archive is a public dataset, which is why it belongs with the training crawlers even though the page itself talks about research access.

Apple’s Applebot page (support.apple.com/en-us/119829) says Applebot’s data powers “Spotlight, Siri, and Safari,” that Applebot “respects standard robots.txt directives in general search crawls that are targeted at Applebot,” and that Applebot-Extended is “a secondary user agent” that lets publishers “opt-out from having their content used to train generative foundation models.” The same Apple page says “webpages that disallow Applebot-Extended can still be included in search results,” because the extended token “is only used to determine how to use the data crawled by the Applebot user agent.” The shape matches Google’s, one crawler and one control token for training.

Meta’s web crawlers page (developers.facebook.com/docs/sharing/webmasters/web-crawlers) lists meta-externalagent/1.1, which “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly,” and meta-externalfetcher/1.1, which “fetches individual links at a user’s request” and “may bypass robots.txt rules.” Per Meta’s page, its training crawl and its indexing crawl share the one meta-externalagent token, so a vendor cannot separate the two the way it can with OpenAI’s three.

What a B2B software vendor should allow

Ask of each token whether it has a documented path to a recommendation. Search crawlers and user fetchers do. Training crawlers do not, per the pages read above, and blocking them is a legitimate policy choice with no documented effect on search-grounded answers. Our own position is to allow every one of them. A training opt-out protects text a company wants to sell, and a vendor page exists to be read.

If you sell property software and want to be named, allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, Googlebot, and Applebot, and never return a 403 to any of them. Decide GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, and Meta-ExternalAgent on your training policy, knowing that none of the documents read here says a Disallow on those changes what the search side shows.

Token	Operator	Job per its documentation	robots.txt per its documentation	webpossible.com
GPTBot	OpenAI	Training	Honored	Allow, named
OAI-SearchBot	OpenAI	ChatGPT search index	Honored, about 24 hours to adjust	Allow, named
ChatGPT-User	OpenAI	User-triggered fetch	May not apply	Allow, named
PerplexityBot	Perplexity	Perplexity search index	Allow recommended; no explicit statement	Allow, named
Perplexity-User	Perplexity	User-triggered fetch	Generally ignored	Allow, named
ClaudeBot	Anthropic	Training	Honored	Allow, named
Claude-User	Anthropic	User-triggered fetch	Honored	Allow, named
Claude-SearchBot	Anthropic	Search quality	Honored	Allow, wildcard
Google-Extended	Google	Training and grounding control token	Token only, no crawler	Allow, named
Googlebot	Google	Search index, AI Overviews, AI Mode	Honored	Allow, wildcard
CCBot	Common Crawl	Open crawl archive	Disallow given as the opt-out	Allow, named
Applebot-Extended	Apple	Training control token	Token only, no crawler	Allow, wildcard
Meta-ExternalAgent	Meta	Training and indexing	Subject to rules	Allow, wildcard
Meta-ExternalFetcher	Meta	User-triggered fetch	May bypass	Allow, wildcard

The robots.txt at webpossible.com allows every user agent with User-agent: * and Allow: /, then repeats an explicit Allow: / for nine named tokens, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Google-Extended, and CCBot, and points to the sitemap index; the five other rows in the table are not named in the file and are allowed by the wildcard line. Nothing on the site is disallowed for anyone. The file also says, in a comment, that a Markdown twin of every page sits at the page URL plus index.md and that a curated map is at /llms.txt, which the guide to llms.txt files explains.

Why blocking a training bot does not remove you from search-grounded answers

The operators separate the controls, and the separation is the whole basis for the claim. OpenAI’s bots documentation ties the training opt-out to GPTBot and ties appearance in ChatGPT search answers to OAI-SearchBot, and says nothing that links the two. Google’s crawler documentation says Google-Extended does not affect inclusion in Google Search, and its AI features page says Googlebot’s robots directives are the control for how a site is crawled for Search, AI features included. Apple’s Applebot page says pages that disallow Applebot-Extended can still appear in search results. Anthropic’s article gives a separate Disallow block for each of its three bots and describes the three jobs separately.

The documents stop there. None of them promises that a page blocked from training will be treated identically at answer time, and none of them describes what the model already knows from earlier training. Our hypothesis is that a vendor whose pages were in a model’s training data can be named from memory with no search at all, and that a vendor who blocks training crawlers today gives up that path for future models while keeping the search-grounded one; we have not tested it and will not claim it until we have run prompts with the search tool disabled.

Search-grounded is the path the Index measures. In the 2026-09 edition, on a single maintenance-software prompt captured through the OpenAI API on 3 September 2026, ChatGPT consulted 124 URLs, 38 of them on reddit.com. One run, one engine, one prompt. It is cited here because every one of those 124 pages had to be fetchable to be consulted. A robots rule or a firewall rule that stopped the fetch would have removed the page from the list before the model read anything. How the Index reads consulted lists and citations is on how we measure AI recommendations.

Whether user-triggered fetchers honor robots.txt

Three operators’ documentation says their fetcher may not, and Anthropic’s says its fetcher does. OpenAI’s documentation says robots.txt rules “may not apply” to ChatGPT-User because the actions are initiated by a user. Perplexity’s documentation says Perplexity-User “generally ignores robots.txt rules.” Meta’s documentation says Meta-ExternalFetcher “may bypass robots.txt rules.” Anthropic’s article says its bots, Claude-User among them, honor robots.txt directives.

The practical reading for a vendor is that robots.txt is the wrong tool for controlling fetchers and the right tool for controlling crawlers. A fetch happens because a property manager asked an assistant about maintenance software, or pasted your pricing URL. Allowing it costs one page load. Blocking it at the server means that person’s assistant reports that your page could not be read, and the assistant answers from whatever else it found, which in the run above was mostly a forum.

Verification in server logs

Documentation says what a bot is supposed to do. Logs say what reached you. There are four checks, in order.

First, filter access logs on the token names: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot, Googlebot, Applebot, CCBot, meta-externalagent, meta-externalfetcher. Match on the token, not the full string, because the version numbers and the browser prefix change. Anthropic’s article prints no full string at all.

Second, check the source IP against the published lists where a list exists. OpenAI, Perplexity, and Anthropic each publish a JSON file, named in their sections above. A line that carries the GPTBot token from an address outside openai.com/gptbot.json is someone else wearing the name, and it should not inform your policy either way.

Third, read the status codes per bot. A 200 is a page read. A 403 is a refusal, and it is the number to hunt for, because most 403s to AI bots are not written in robots.txt. They come from a CDN bot-management setting, a firewall rule, or a rate limiter that matched the user-agent. Check those settings before assuming your robots.txt is the whole policy. A 301 chain or a 5xx on the pricing page is worth the same attention.

Fourth, count fetches by path. If Perplexity-User and ChatGPT-User are opening your pricing and integrations pages, people are asking assistants about you and the assistants are looking. If the fetchers only ever hit the homepage, the questions being asked do not reach the pages that carry the facts, and that is a content problem covered in the AI visibility guide.

What a 403 to a search bot costs is stated most directly by OpenAI. Per OpenAI’s bots documentation, sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. A firewall 403 is not the same signal as a robots Disallow, and the OpenAI sentence is about the Disallow, so we do not claim the 403 produces the identical outcome. What a 403 does, without dispute, is deny the page to the crawler on that request, and a page that cannot be read cannot be indexed from that read. For a fetcher, the cost is the single answer in front of the single person who asked. That answer is where the recommendation is made, and the audit of what AI says about a vendor starts by reading these same log lines.

Limits

This page is documentation, read on one day. Every quoted sentence comes from a page named in the sentence, and every one of those pages can change without notice. Nothing here reports what any bot did on any particular site beyond the one Index run cited above.

The documents describe intent, not enforcement. A statement that a bot honors robots.txt is the operator’s statement. Only your logs can confirm it for your domain.

Robots.txt is advisory. It is read by bots that choose to read it, and the operators above say their fetchers may not. A vendor that needs a hard block needs a server rule, and a vendor that wants to be recommended should not need one.

The table reflects webpossible.com’s file, which is a publisher’s choice. A vendor with a different view of training can disallow GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and Meta-ExternalAgent and, per the documents above, keep every search and fetch path open. That is a defensible file. Blocking OAI-SearchBot, PerplexityBot, or Googlebot is a different decision with a documented cost.

Structured data and the Markdown twin do not change any of this. A page must be reachable before markup on it matters; the structured data guide assumes the fetch succeeded.

What changes month to month

Tokens and versions. OpenAI’s bots page carried 1.4 for GPTBot and OAI-SearchBot on the day of reading, and the browser prefix on OAI-SearchBot names a Chrome version. Match on tokens in any filter you keep.

Robots policy per fetcher. The three operators whose fetchers may ignore robots.txt could change that position, and Anthropic could change its own. Re-read the four documents before repeating the table.

The IP lists. The JSON files are the source of truth for whether a request is genuine, and they rotate. A verification script should fetch the list each run rather than cache it.

Google’s AI features. Google’s AI features documentation says Googlebot’s robots directives and the snippet controls are the controls for AI in Search; if Google ever introduces a separate token for those features, this page’s Google section is wrong from that day.

The documentation URLs. OpenAI’s bots page moved from platform.openai.com to developers.openai.com and Anthropic’s help center moved from support.anthropic.com to support.claude.com before 2026-09-04, and both old URLs redirect. Cite the current ones.

Related guides

The engine pages read the answer side of what these bots feed: the ChatGPT source-selection guide for the app capture and what the links in an answer leave out, the Perplexity citation guide for dated results and numbered citations, the AI Overviews guide for the answer box Googlebot governs, and the guide to how Claude picks vendor sources for Anthropic’s fetch tool. The guide to llms.txt files covers the map file this site’s robots.txt comments point to, and the structured data guide covers what to put on the page once the fetch succeeds. Metric definitions live in the AI visibility guide, the Index’s protocol is on how we measure AI recommendations, and the pillar on getting named by AI assistants puts the pieces in order.

Changelog

• 2026-09-04: First published.