How to block AI bots: three layers, and where a block also stops search crawlers

Nov 9, 2026 · 9 min read

You can block AI bots at three layers: a line in robots.txt that asks a named crawler to stay away, an edge or server rule that refuses the request, and a setting for the pages an assistant fetches because a person asked it to, which the assistants' owners say the file may not govern. The layer you pick decides what else you lose. Since 15 September 2026, Cloudflare's own Block setting for training also stops Googlebot.

Everything here comes from the companies' own pages; we ran no experiment. Our interest is plain: the software we build logs the sources that AI answers cite, and blocking is something we have never measured.

We reviewed the top-ranking pages on 3 November 2026: Cloudflare's, Netlify's and Fastly's documentation, Raptive's advice to the publishers in its network, and the ai.robots.txt project's shared list. Four of those five are written by a vendor or a network about its own product or members, and each serves that reader well. Only Cloudflare's page carries the change it made on 2 November.

What are the three ways to block AI bots?

Robots.txt, an edge or server rule, and a setting for the agents a person sends are the three ways to block AI bots. Each one reaches a different set of AI bots.

Robots.txt: the file asks, and a crawler decides whether to listen. The policies page Cloudflare publishes says so in one line: "robots.txt directives are voluntary. Crawlers can ignore them." Raptive tells its members that editing the file is "nicely requesting that bots don’t take your content." The per-bot lines, and what each owner says they mean, are on our page about each AI training crawler.

An edge or server rule: the request is refused before your page is served, usually by a CDN in front of your server or by the server itself. Raptive's post notes that "Many CDNs also offer security features including bot detection and blocking." Netlify's guide builds an edge function that returns "an HTTP 401 response" to any AI bot on its list, and warns that "some bots have been found to mis-report their names in their User Agent strings". The ai.robots.txt project ships its list as a robots.txt file and as `.htaccess`, nginx, Caddy, HAProxy and lighttpd rules. Fastly, whose post describes its own edge tools, states the limit plainly: "Even Fastly’s system can’t distinguish between AI bots that ignore robots.txt and those that follow it."

The fetches a person triggers: when someone asks an assistant to read your page, the request comes from a user agent its owner treats apart from its crawlers, and OpenAI, Perplexity and Anthropic each give a different answer on whether the file applies. These fetches are a kind of request, not a third mechanism: where the file may not reach them, refusing them falls to an edge or server rule, such as an Agent setting or one that names the fetchers.

What changed in Cloudflare's settings on 2 November?

Cloudflare's AI bot control has a new name, and its 2 November 2026 update says new domains start with every AI behaviour allowed. The documentation page, AI Crawler and Agent access policies, carries the stamp "Last updated Nov 2, 2026" and a note: "This feature was previously called Configure AI bot policies."

The policies "let you control AI crawler and agent traffic by behavior." Search covers "crawlers that collect or index your content to answer questions about it later". Agent is "automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents". Training covers crawlers that train or fine-tune a model, "including mixed-purpose crawlers that are used both for Training and for Search".

Each behaviour takes one of four actions: Block, Block on pages with ads, Allow, and Disallow, which is offered only for Training: "For Training only, tell crawlers not to access your content through robots.txt". With Bot Preference Sync on, Disallow also refuses training crawlers at the edge, and mixed-purpose crawlers "are excluded from this edge block."

The new-domain default has moved between two dated statements. The bots changelog entry of 1 July 2026 let customers manage AI crawlers by these behaviours "instead of a single Block AI bots toggle". The same entry said new defaults, with Training and Agent blocked on pages that display ads, would "take effect for new domains on September 15, 2026." The policies page as updated on 2 November says: "For new domains, Search , Agents , and Training use Allow (do not block) by default." The changelog had no entry for that change when we read it, and neither page gives a reason.

Personal Agents, the named assistants a person uses, inherit the Agent setting, and "Cloudflare recommends Allow (do not block) for Personal Agents." That is Cloudflare's recommendation, not ours.

Where does a block also stop search crawlers?

A Training block at Cloudflare and a shared list of AI bots copied whole both stop search crawlers. Neither is a fault in the tool; each does what its owner documents.

The reach of the Training actions widened in September. Cloudflare's post of 15 September 2026 says Block and "Block on pages with ads" "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training." The setting it added to let those crawlers keep crawling for search, and its paid-access options, are covered on our Pay Per Use page.

A shared list reaches further than training by design. The ai.robots.txt README says its list "contains AI-related crawlers of all types, regardless of purpose", and adds: "We encourage you to contribute to and implement this list on your own site." On 3 November 2026 its robots.txt carried 202 user-agent lines, among them four search crawlers whose owners document them:

• OAI-SearchBot surfaces websites in ChatGPT's search features; a site that opts out is not shown in ChatGPT's search answers, though a navigational link may still show (OpenAI).
• PerplexityBot surfaces and links websites in Perplexity's search results (Perplexity).
• Claude-SearchBot navigates the web to improve Claude's search results, and disabling it may reduce a site's visibility and accuracy in them (Anthropic).
• Applebot's crawl powers search in Spotlight, Siri and Safari, and allowing it lets a site appear in those results (Apple).

Two of the ranking pages already warn about Google's crawler, and their advice holds as far as it goes. Raptive, on 8 April 2026: "we don’t recommend blocking Googlebot (or Google-extended) because the cost to your search visibility is too high." Fastly, in a post of 19 August 2025 updated on 2 September 2026: "Google-Extended isn’t a bot. It’s a flag." What the dated settings add is that the Googlebot decision can now be taken by a dashboard setting rather than by a line you wrote. What Google itself documents for Google-Extended is a page of its own.

Can you block an AI bot a person sends?

Robots.txt reaches the fetches a person triggers in some assistants and not in others, by each AI company's own account. OpenAI says ChatGPT-User may visit a page when someone asks ChatGPT a question, and "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity's crawler page, modified on 29 October 2026, says of Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Anthropic's help page of 7 April 2026 puts its own fetcher inside the file's reach: "Claude-User allows site owners to control which sites can be accessed through these user-initiated requests." Anthropic adds that disabling Claude-User "may reduce your site's visibility for user-directed web search."

The edge is the layer that can refuse these requests whatever a file says, and Cloudflare's Agent behaviour is built for them. Perplexity's page works the other way round, telling sites behind a web application firewall how to let its bots through and to "match each User-Agent against its own IP ranges". Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because doing so "impedes our ability to read your robots.txt file".

We think that when a site refuses an assistant's fetcher, the assistant cannot quote the page from that fetch. That is our reading of how the fetch works, not something we have measured.

How do you choose which layer to block AI bots at?

The right layer for blocking AI bots depends on what you want to stop. Each goal points at a different one:

• To state a preference to crawlers that honour it: a line in robots.txt under each user agent.
• To refuse a crawler that ignores the file or uses another bot's name: an edge or server rule that checks both the user agent a request claims and whether its source address sits inside the IP ranges the named owner publishes.
• To ask crawlers not to train on your pages while staying in search on Cloudflare: Training set to Disallow rather than Block, since Block now reaches Googlebot. For mixed-purpose crawlers such as Googlebot, Disallow is a request in robots.txt, not a refusal at the edge.
• To decide about the fetches a person triggers: the Agent behaviour, or a rule naming those fetchers, weighed against what each AI company says about refusing them.

Three checks before you change a setting; these are our suggestions, not a standard. Open your dashboard and note what each AI behaviour is set to today, and what a new domain now starts on. Read every name on a copied list against its owner's page before you deploy it. Check your logs against the IP ranges the owners publish, because a bot can claim another bot's name.

None of these layers is the right one for every site. Which costs your site can carry in search and in AI answers is your decision, and we leave it with you.

Sources

• Cloudflare, AI Crawler and Agent access policies (last updated 2 November 2026; checked 3 November 2026) — https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/
• Cloudflare, Bots changelog, entry of 1 July 2026 (checked 3 November 2026) — https://developers.cloudflare.com/bots/changelog/
• Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training" (15 September 2026; checked 3 November 2026) — https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
• OpenAI, Overview of OpenAI Crawlers (checked 3 November 2026) — https://developers.openai.com/api/docs/bots
• Perplexity, Perplexity Crawlers (modified 29 October 2026; checked 3 November 2026) — https://docs.perplexity.ai/docs/resources/perplexity-crawlers
• Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (7 April 2026; checked 3 November 2026) — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Apple, About Applebot (checked 3 November 2026) — https://support.apple.com/en-us/119829
• Netlify, "Blocking AI bots and controlling crawlers", by Phil Hawksworth (checked 3 November 2026) — https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/
• ai.robots.txt project, README and robots.txt (read as they stood on 3 November 2026) — https://github.com/ai-robots-txt/ai.robots.txt
• Raptive, "How blocking AI bots paves the way for fair compensation" (8 April 2026; checked 3 November 2026) — https://raptive.com/blog/ai-bot-blocking-strategy/
• Fastly, "The Truth About Blocking AI, And How Publishers Can Still Win", by Simon Wistow (19 August 2025, updated 2 September 2026; checked 3 November 2026) — https://www.fastly.com/blog/the-truth-about-blocking-ai-and-how-publishers-can-still-win
• The ranking pages we reviewed for "block ai bots" (US results, 3 November 2026): developers.cloudflare.com, developers.netlify.com, github.com/ai-robots-txt, raptive.com and fastly.com.