Two identical royal-blue bars on one baseline, the left one wearing an orange cap fitted over its top end, the cap's top level with the right bar's top; beneath them the words SAME CRAWL and, smaller, MODEL TRAINING.

Does blocking Google-Extended affect AI Overviews? What Google documents, and what the studies measured

Nov 1, 2026 · 11 min read

Google documents Google-Extended as a robots.txt control over Gemini training and grounding that does not affect whether a site is included in Google Search. For AI Overviews, the documentation names other controls: a Search Console setting and the `nosnippet` rule. In one peer-reviewed study of Google results collected in December 2025, sites that blocked the token were less likely to be among AI Overview sources than among ordinary results; its authors call their analyses descriptive.

This page sets the documented controls side by side, reads that study and two vendor counts, and leaves the decision with you. We build software that tracks AI citations, and we have measured nothing about the token ourselves.

What does Google-Extended control, in Google's words?

Google-Extended controls whether content Google crawls may train Gemini models and ground Gemini's answers, and it is a token you name in robots.txt, not an AI crawler with its own visits to your server. Google's list of common crawlers says it "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Googlebot and the company's other crawlers still crawl your pages; the token decides what may be done with what they collect.

The same entry says what that is. Publishers can use it "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding" in Gemini Apps and on Vertex AI. Most pages on this question repeat the entry's next sentence: the token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

The page is stamped 15 October 2026, and archived copies show its text on the token unchanged since at least the 14 July 2026 revision. The entry does not mention AI Overviews or AI Mode at all. Our earlier page on generative engine optimization strategies quotes the same line.

Disallowing the token takes two lines in your robots.txt file:

``` User-agent: Google-Extended Disallow: / ```

Which controls does Google document for AI Overviews?

There are three documented ways to decide if content appears in the AI features of Search, and the token is not one of them: a Search Console setting, the snippet rules, and `noindex`. Crawling for Search itself is governed by your robots.txt rules for Googlebot.

Under the heading AI OVERVIEWS, three identical plates reading SEARCH CONSOLE, NOSNIPPET and NOINDEX; set apart on the right under TRAINING AND GROUNDING, one orange-outlined plate reading GOOGLE-EXTENDED. — Google's documentation names three controls for whether content appears in AI Overviews: the Search Console setting, nosnippet and noindex. Google-Extended is not among them; it governs the training and grounding of Gemini, and the training of the models behind Search's AI features.

Search Console's Search generative AI control covers AI Overviews, AI Mode and generative AI features in Discover. The help page says that "As of August 31, 2026", the setting had been rolled out to all websites worldwide. Excluding a site means its content is not shown or linked in those features and does not ground their answers, and "You won't receive any traffic or impressions from these features." The page says the setting "isn't used as a ranking or inclusion signal affecting other parts of Search", and that a change usually takes a few days, with some content delayed longer by caching. The choice is made per property, so a child property, such as a URL-prefix property, inherits its parent's choice unless its owner overrides it.

That help page is the only place we found where the documentation ties the token to the AI features in Search, and only for model training: "This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended." The sentence is about which content may train the models. It does not say the token decides whether a page appears in AI Overviews.

The second way is the robots meta tag. The meta tag documentation says `nosnippet` applies to every kind of result, AI Overviews and AI Mode included, and "will also prevent the content from being used as a direct input for AI Overviews and AI Mode." The cost is that the specific page also loses its text snippet in ordinary results. `max-snippet` limits how much of the content can be drawn on. These rules can be read only if crawlers are allowed to reach the page.

AI features and your website puts crawling plainly: "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." How Search then picks among the pages it may show is a separate question, which our page on how Google chooses the sources an AI Overview shows covers.

Side by side, as the documentation describes them:

Google-Extended in robots.txt: limits Gemini training and grounding in Gemini Apps and Vertex AI, and the training of models behind the generative features in Search. Documented as not affecting inclusion or ranking in Search.
The Search generative AI control: removes a site's links and content from AI Overviews, AI Mode and Discover's generative features. Documented as no ranking or inclusion signal elsewhere in Search, and as not affecting training.
`nosnippet` and `max-snippet`: stop or limit a specific page's content as a direct input to AI Overviews and AI Mode, along with its snippets in every result.
`noindex`: keeps a page out of Search entirely, AI features included.

What did the SIGIR 2026 study measure?

In search results collected on 7 and 8 December 2025, sites that disallowed Google-Extended were less likely to be among AI Overview sources than among ordinary results, after controls for popularity and site type; it is an association in observational data. The paper is How Generative AI Disrupts Search, by Riley Grossman, Songjiang Liu, Michael K. Chen, Mike Smith, Cristian Borcea and Yi Chen, posted on 30 April 2026 and accepted at ACM SIGIR 2026.

The query set holds 11,500 searches in nine groups, among them real user queries, shopping searches, debate questions, local "near me" searches and questions from a public research set. The full text says each query went through SerpAPI, "instructed to simulate a mobile device from Newark, NJ", to capture the ordinary results and any AI Overview, and through the Gemini 2.5 Flash model with search grounding switched on. The authors call AI Overviews and Gemini generative search engines and compare each with the traditional search engine results.

The finding started with Gemini. In that data, 21 popular publishers that Search and AI Overviews each retrieved for at least 20 queries were never cited by Gemini, and all 21 blocked the token when the authors checked; NYTimes, BBC, Reuters and Nature are among them. The paper then adds that "many of these websites also have fewer citations in AIOs", naming NYTimes, Yelp, IMDb and Facebook.

The wider test is a linear probability model with query fixed effects across 165,934 pairs of a source and a query, covering every source at least one of the three engines retrieved. It controls for a site's popularity on the Tranco ranking, 17 Cloudflare site categories and an `.edu` address. Across those 165,934 pairs, the estimate for AI Overviews in the regression output published with the authors' code is −0.136, with a 95% interval of −0.151 to −0.120. In plain terms, after the controls, a source's chance of appearing in AI Overviews, measured against its chance in the ordinary results, was about 14 percentage points lower when its site disallowed the token than when it did not. For Gemini the same estimate is about −0.60.

The authors draw their own conclusion: "publishers may need to rethink their decision to block Google from accessing their content for AI training and grounding."

What does the study not show?

Cause is beyond what the SIGIR paper can establish: nothing in it shows that disallowing Google-Extended produced the lower AI Overview figure. Its authors describe their analyses as "both descriptive in nature and limited to a single point in time."

Two plates, one reading TOKEN BLOCKED and one reading AI OVERVIEWS LESS LIKELY, with a plain orange line between them and no arrowhead; beneath, WENT TOGETHER, CAUSE NOT SHOWN and, smaller, DECEMBER 2025. — In the SIGIR study's search results from two days in December 2025, sites that blocked the token were less likely to be among AI Overview sources than among ordinary results, after the study's controls. The two went together; nothing in the paper shows that blocking caused it.

The sites that opt out are not a random set. Large news publishers, social networks and review sites differ from other sites in many ways, and a popularity rank and a site category may not catch all of them.

The paper is silent on when and how each site's robots.txt was read, and on what counted as a block. That matters for any case where a site changed its file around the collection dates.

The main collection came from a simulated phone in one US city, on Google only; the authors' robustness checks on two 10% samples, run as desktop searches from other locations, do not report the Google-Extended result separately. The data also predates the Search Console setting's rollout to every site in August 2026, so the paper could not see it. The authors wrote that Google allowed sites to leave AI Overviews only by leaving Search; the documentation now describes the Search Console route as well.

Nothing in the paper, and nothing in the documentation, says the token decides which pages appear in AI Overviews. The authors measured what went together in those two days of results, not the mechanism behind it.

What do the vendor counts add?

Sites that disallow Google-Extended still appear in AI answers, according to two vendor counts, but that answers a different question from whether they appear as often. Neither count compares opted-out sites with sites that allow the token.

BuzzStream's count, by Vince Nero and updated on 8 April 2026, drew on "4 million citations from 3,600 prompts in ChatGPT, Gemini, AI Overviews, and AI Mode" from Citation Labs' XOFU tracking tool. Most of the top news sites in its sample that disallow the token still appeared somewhere in those citations. BuzzStream does not give the number of such sites, and it pools the four products, so AI Overviews cannot be read on their own.

The New York Times drew zero Gemini citations and 8,007 AI Overview citations among the 31 million AI citations Goodie recorded between October 2025 and July 2026, in a report published on 28 July 2026. Goodie says the Times refuses all 13 major AI user agents; it read the robots.txt files of 105 US and UK news publishers. That is a single publisher's count, and it was published before the Search Console setting reached every site.

The counts and the controlled comparison fit together. A site can keep appearing in AI Overviews and still appear less often than it does in ordinary results. Of the three measurements we found in searches run on 19 October 2026, the SIGIR paper is the only one that compares opted-out and other sites with controls. BuzzStream sells outreach software, and Goodie sells AI-visibility monitoring.

What do the pages that rank for this question say?

We read six of the eight pages that ranked for this question in US results on 19 October 2026, and not one cites a measurement of opting out and AI Overviews. Most are SEO and marketing blogs explaining the token.

AIGROW's page files AI Overviews under Googlebot and says the token "does not control Google Search rankings". Altimizo describes it as an opt-out for "some Google AI training uses". Clearscope's 2024 article points to `nosnippet` as the way to keep content out of AI Overviews. GetCito argues the other way for AI answers in general: that "blocking content from training LLMs results in your brand becoming invisible in generated AI answers".

How do you decide whether to block it?

Whether to block Google-Extended depends on your reason for blocking it, and the reasons are yours: limiting how AI models learn from your pages, a licensing position, or keeping Gemini from grounding answers in your pages. We do not recommend either choice.

If your reason is training or Gemini, the token is the documented control for it, and the SIGIR paper is the one controlled measurement we found of a possible cost in AI Overviews, with the limits above. If your goal is to stay out of AI Overviews, the documentation points to the Search Console setting and `nosnippet` for that, not the token, and each has the costs described above.

Before you decide, four checks, which are our suggestions rather than a standard:

Open the file at your domain's /robots.txt URL and see whether a `User-agent: Google-Extended` group is there. You may not have written it: the introduction to robots.txt notes that a CMS may manage the file for you, and an SEO plugin or a CDN setting can add rules too, as Cloudflare's crawler setting of 15 September 2026 does.
Note the date of any change, for comparison later.
Keep monitoring Search Console's Generative AI performance report in the weeks after; Google offers it to measure performance in those features.
Remember that robots.txt is a request, and a crawler can ignore it. The same page says that "Googlebot and other respectable web crawlers obey the instructions", while "other crawlers might not". The token sits beside other AI crawler names, such as GPTBot, OpenAI's training crawler, and each needs its own line.

Questions people ask

Does blocking Google-Extended stop Googlebot from crawling my site?

No. Google-Extended has no user agent of its own; crawling uses the existing user agent strings, and the token is read "in a control capacity". A `Disallow` rule under that token does not stop Googlebot.

Will blocking Google-Extended hurt my rankings?

Google-Extended is not a ranking signal in Search, according to the crawler documentation, and does not affect a site's inclusion there. The SIGIR paper looked at which sources AI Overviews and Gemini drew on, not at rankings.

Can I keep my pages out of AI Overviews and stay in search results?

Yes, in two documented ways. The Search Console setting removes a whole site or property from AI Overviews, AI Mode and Discover's generative features; `nosnippet` removes a specific page's content as a direct input, at the cost of its snippet in ordinary results.

Sources

"Google's common crawlers", the Google-Extended entry, crawling infrastructure documentation (page stamp 15 October 2026; read 19 October 2026) — https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
The Search Console Help page on the "Search generative AI control" (available to every website from 31 August 2026; read 19 October 2026) — https://support.google.com/webmasters/answer/16908024
Robots meta tag, data-nosnippet and X-Robots-Tag specifications, Search Central (page stamp 24 March 2026; read 19 October 2026) — https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
The AI features page for site owners, Search Central (page stamp 10 December 2025; read 19 October 2026) — https://developers.google.com/search/docs/appearance/ai-features
The robots.txt introduction, Search Central (page stamp 10 December 2025; read 19 October 2026) — https://developers.google.com/search/docs/crawling-indexing/robots/intro
Riley Grossman and five co-authors, "How Generative AI Disrupts Search", arXiv 2604.27790, first version posted 30 April 2026, accepted at ACM SIGIR 2026 (read 19 October 2026) — https://arxiv.org/abs/2604.27790 · full text https://arxiv.org/html/2604.27790v1
The same authors' regression notebook, SIGIR_Regressions.ipynb, in the rag24/AIO repository (read 19 October 2026) — https://github.com/rag24/AIO/blob/main/scripts/SIGIR_Regressions.ipynb
Vince Nero for BuzzStream, "Do News Publishers That Block AI Crawlers Get Cited Less Often by AI?" (updated 8 April 2026; read 19 October 2026) — https://www.buzzstream.com/blog/news-block-ai-bots-citations/
Goodie, "AI Citations & News Publishers: 2026 Study" (published 28 July 2026; read 19 October 2026) — https://higoodie.com/blog/publishers-and-ai-search-study/
The ranking pages we read for "does blocking google-extended affect ai overviews" (US results, 19 October 2026): a LinkedIn post by Terry Robinson, webiano.digital, altimizo.com, aigrow.gg, clearscope.io and getcito.com.