Can robots.txt block AI training? What the file controls, what it cannot, and the usage rule being drafted
Partly. Robots.txt can keep an AI crawler from fetching your pages, as long as that bot honours the file, but on its own it does not decide what happens to content already collected. Google's Gary Illyes said as much on 21 October 2026, and the rule that would let a site state a training preference in the file is still an IETF draft.
This page sets Google's words beside what OpenAI, Anthropic, Common Crawl, Apple and Perplexity each document for their crawlers, and reads that draft. It is a read of documents, not a test: we build software that tracks the sources AI answers cite, and we have measured nothing about training opt-outs ourselves.
We reviewed seven of the top-ranking pages for this question. All seven describe keeping an AI crawler out by naming its user agent in robots.txt, and that advice holds as far as it goes. None of them mentions Google's statement or the draft.
What does robots.txt actually control?
Robots.txt controls which URLs a crawler may request, and the standard that defines the file says nothing more than that. RFC 9309, the Robots Exclusion Protocol, was published on the IETF's standards track in September 2022; its rules are ones "that crawlers are requested to honor when accessing URIs", and "These rules are not a form of access authorization."
Gary Illyes of Google drew the line for AI on 21 October 2026, in the second episode of Google Search Central's The Non-Canonical Cabinet, What can you actually put in a robots.txt?, recorded with Cherry Prommawin. As transcribed by Search Engine Roundtable, he said: "So, as defined by RFC 9309, robots.txt is an access-preference protocol." And: "It doesn't dictate what a system does with the data after it has already crawled it." A disallow rule, in his words, "bluntly means, "Do not enter the room.""
His practical advice was narrow. To stop one AI crawler from reaching your pages, "you have to target its specific user-agent token explicitly in the file." A line the parser does not understand is silently discarded, so an invented "no AI training" instruction does nothing. Blocking OpenAI's GPTBot from a whole site takes two lines:
``` User-agent: GPTBot Disallow: / ```
How do you block each AI training crawler?
A separate user agent group for each bot, with `Disallow: /` under it, keeps each documented training crawler out, and each owner's page says what the block means. For OpenAI and Anthropic, the block is also a statement about training.
GPTBot (OpenAI): GPTBot collects content that may be used to train OpenAI's foundation models, and disallowing it "indicates a site's content should not be used in training generative AI foundation models". OAI-SearchBot is a separate user agent token for ChatGPT's search features, so a site can allow OAI-SearchBot and block GPTBot. When a site allows both bots, OpenAI "may use the results from just one crawl for both use cases." ChatGPT-User fetches pages when a person asks, and for it "robots.txt rules may not apply."
ClaudeBot (Anthropic): restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets", according to Anthropic's help page of 7 April 2026. Claude-User and Claude-SearchBot are separate bots, and the block has to be added for every subdomain.
CCBot (Common Crawl): the CCBot user agent token is blocked with `User-agent: CCBot` and `Disallow: /`, and Common Crawl warns that some bots falsely identify themselves as CCBot.
Applebot-Extended (Apple): Applebot-Extended "does not crawl webpages"; it only decides how data crawled by Applebot is used, according to Apple's page of 4 September 2026. Disallowing it opts a site out of training Apple's foundation models, and its pages can still appear in Apple's search results. Apple documents a second control outside robots.txt: the `nosnippet` robots meta tag, or the same rule in an X-Robots-Tag header, tells Apple not to use specific content as context for the AI-generated answers in its products, though it is not a training opt-out.
PerplexityBot (Perplexity): PerplexityBot is a search crawler, and Perplexity says "It is not used to crawl content for AI foundation models." Blocking it is not a training opt-out.
Applebot-Extended is a rule about use rather than a crawler. Google-Extended is another: Google's list of common crawlers calls it a token "used in a control capacity" for Gemini training and grounding, and what Google documents for that token is a page of its own. As an example of the syntax only, a file that names four of these user agents looks like this:
``` User-agent: GPTBot Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: Applebot-Extended Disallow: / ```
Does blocking a crawler keep your content out of AI training?
Blocking a training crawler stops that bot collecting new copies of your pages, from that point on and only if it honours the file. The documents we read promise nothing about content collected before a block, or about copies other people hold.
Illyes's line covers the first part: the file does not govern what happens to data already crawled. Anthropic's word is "future". OpenAI's disallow "indicates" a preference; it does not describe removing anything.
Common Crawl calls itself a non-profit "providing a copy of the Internet to Internet researchers, companies and individuals at no cost", and its crawl data can be "bulk downloaded". A robots.txt disallow means "our crawler will stop crawling your website", in its FAQ's words. None of the Common Crawl pages we read says a block removes pages from crawls it has already published.
The IETF draft discussed below agrees on timing: updates to robots.txt "do not retroactively apply to resources", so a changed preference applies only after a bot fetches the new file and the page again.
Two of the seven ranking pages say the same in their own words. SerpSurfers writes that robots.txt "does not remove content that has already been collected", and WebDecoy that "You can't retroactively remove your content from training datasets."
A search crawler is a different decision. Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links", in OpenAI's words, while blocking GPTBot leaves OAI-SearchBot untouched.
Do AI training crawlers obey robots.txt?
GPTBot and ClaudeBot obeyed a robots.txt file that disallowed every page in a controlled 2025 study, but other bots in the same study ignored the file or never read it. The standard itself only requests compliance, and Anthropic's help page states that its bots "respect "do not crawl" signals by honoring industry standard directives in robots.txt".
Scrapers selectively respect robots.txt directives, by Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay and Emily Wenger of Duke University, was revised on 23 October 2025. Its sample is about 3.9 million requests to 36 university websites between 12 February and 29 March 2025, from 130 self-declared bots. Three increasingly strict files ran on a single site for two weeks each. Bots "are less likely to comply with stricter robots.txt directives", the authors found. Under the file that disallowed every page, GPTBot and ClaudeBot scored as fully compliant and ByteDance's Bytespider scored zero. 15 of 47 bots never checked robots.txt during the two stricter tests. The sample is one US university's sites in early 2025, and the authors attribute some non-compliance to bots faking another bot's user agent. It is one study among the few we found.
Robots.txt asks; a server block enforces. Cloudflare's robots.txt setting documentation puts it plainly: "use robots.txt to express your preferences and AI Crawl Control to enforce crawler-specific blocks". The same split runs through Cloudflare's own blocking and paid-access settings.
What is the Content-Usage rule being drafted?
A proposed robots.txt line and HTTP header called Content-Usage would let a site state how crawled content may be used; it is still an Internet-Draft, not a standard. It would sit beside the rules on what may be crawled rather than replace them.
draft-ietf-aipref-attach-05, "Associating AI Usage Preferences with Content in HTTP", is by Gary Illyes of Google and Martin Thomson of Mozilla, dated 19 August 2026. It is a working-group document of the IETF's AI Preferences group, it has not yet gone to the IETF's steering group for review, and it expires on 20 February 2027 unless revised. If approved, it would update RFC 9309.
The draft adds a `Content-Usage` rule to robots.txt and a `Content-Usage` HTTP header, "a two-stage arrangement that distinguishes acquisition and usage". Preferences "apply only to those resources that can be crawled", so a page you disallow carries no usage preference at all. The draft's own example allows crawling and says no to training:
``` User-Agent: * Allow: / Content-Usage: train-ai=n ```
The words in that line come from draft-ietf-aipref-vocab-08, by Paul Keller and Martin Thomson, dated 14 September 2026. It defines AI training as "Using an asset to modify the learned parameters of a generative AI model". Its note to readers says its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part", and it does not "ensure that preferences are followed". None of the crawler pages we read mentions a Content-Usage line.
A similar line already exists outside the IETF. Cloudflare can write a content signal such as `ai-train=no` into a site's robots.txt, directives its documentation says "categorize how crawlers may use your content", and it is a line that can disagree with the rest of your site.
How do you decide what to put in your file?
What belongs in your robots.txt depends on what you want to stop: a crawler's visits, a company's training use, or your appearance in an AI search product. Those goals do not map one to one onto lines, because the same Disallow does a different job under each name. Under GPTBot or ClaudeBot it asks a crawler to stay out and, in their owners' words, signals a training preference. Under Applebot-Extended or Google-Extended it stops no crawler and governs only use, for Google-Extended both Gemini training and grounding. Under OAI-SearchBot it keeps a site out of ChatGPT's search answers, though the site can still appear as a navigational link. A content signal states a preference and blocks nothing. We do not recommend blocking or allowing any of these lines.
Four checks before you change anything, offered as our view, not a standard:
- Read the robots.txt at your own domain and list every AI user agent it names. A CMS, a plugin or a CDN setting may have written lines you did not.
- For each user agent token, read what its owner says it governs (training, search or a person's request) before you keep or remove it.
- Read your server logs for the user agents you named, and verify them against the owner's published IP ranges or reverse DNS. OpenAI, Common Crawl, Apple and Perplexity publish them, and Common Crawl warns that a user agent can be faked.
- Remember that the file speaks to well-behaved crawlers, from the next visit on. What was collected before sits outside it.
Which of those lines your site needs is your call.
Sources
- Search Engine Roundtable, "Google On robots.txt Rules For AI Crawlers, Emojis, International URLs & More", with the episode's transcript (published 21 October 2026; checked 26 October 2026) — https://www.seroundtable.com/google-video-robots-txt-rules-42355.html
- Google Search Central, "What can you actually put in a robots.txt?", The Non-Canonical Cabinet (checked 26 October 2026) — https://www.youtube.com/watch?v=-jHONLQgE-s
- RFC 9309, Robots Exclusion Protocol (September 2022; checked 26 October 2026) — https://www.rfc-editor.org/rfc/rfc9309.html
- OpenAI, Overview of OpenAI Crawlers (checked 26 October 2026) — https://developers.openai.com/api/docs/bots
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (dated 7 April 2026; checked 26 October 2026) — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Common Crawl, CCBot (checked 26 October 2026) — https://commoncrawl.org/ccbot
- Common Crawl, Frequently Asked Questions (checked 26 October 2026) — https://commoncrawl.org/faq
- Apple, About Applebot (published 4 September 2026; checked 26 October 2026) — https://support.apple.com/en-us/119829
- Perplexity, Perplexity Crawlers (checked 26 October 2026) — https://docs.perplexity.ai/guides/bots
- Google, Google's common crawlers (last updated 15 October 2026; checked 26 October 2026) — https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- IETF Datatracker, draft-ietf-aipref-attach-05 (19 August 2026; checked 26 October 2026) — https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/
- IETF Datatracker, draft-ietf-aipref-vocab-08 (14 September 2026; checked 26 October 2026) — https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/
- Taein Kim and five co-authors, "Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study", arXiv 2505.21733 (version 2, 23 October 2025; checked 26 October 2026) — https://arxiv.org/abs/2505.21733
- Cloudflare, robots.txt setting (last updated 22 October 2026; checked 26 October 2026) — https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
- The ranking pages we reviewed for "can robots.txt block ai training" (US results, 26 October 2026): blog.zgp.org, agentready.md, webdecoy.com, ecentic.ai, junian.dev, serpsurfers.com and iliassami.com.
