Skip to content

All articles

--- title: "GPTBot: what OpenAI's own documentation says it does on your site" primary_keyword: "gptbot" ---

A slack orange cord hangs between the tops of two short blue posts on bare ground; beneath it the words DISALLOWING GPTBOT and INDICATES.

GPTBot: what OpenAI's own documentation says it does on your site

· 4 min read

GPTBot is OpenAI's web crawler for content that "may be used in training" its generative AI models, per OpenAI's crawler page on 11 November 2026. The crawler page documents three other user agents, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, and states "each setting is independent of the others", so a robots.txt line against the training crawler speaks to training alone.

We read the linked page, which shows no last-updated date, and our own robots.txt that same day. We hold no crawler log, so we give no visit figure for any website, ours included. We do not cover gptbot.io, an unrelated prompts and AI-tools site.

What does GPTBot do on my website?

Per OpenAI's page, it is used to crawl content, so a visit means it is fetching pages from your site. Each user agent as listed on OpenAI's crawler page on 11 November 2026:

| User agent | Purpose | What a robots.txt line does | Published IP list | |---|---|---|---| | GPTBot | Content that "may be used in training" the models | "Disallowing GPTBot indicates a site's content should not be used in training" | openai.com/gptbot.json | | OAI-SearchBot | ChatGPT's search features | Opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links" | openai.com/searchbot.json | | ChatGPT-User | May visit a web page on a user's question in ChatGPT or a Custom GPT; "not used for crawling the web in an automatic fashion" | "robots.txt rules may not apply" | openai.com/chatgpt-user.json | | OAI-AdsBot | Pages submitted as ads only; its data "is not used to train" the models | Not stated | openai.com/adsbot.json |

On 11 November 2026 the page labelled this string an "example user-agent string (the version number may change)":

``` Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot ```

OpenAI says that when fetching robots.txt files it may add a `robots.txt` marker to the user-agent string, which helps site owners tell those requests apart in logs.

On OpenAI's crawler page as read on 11 November 2026, GPTBot's visit frequency is not stated; we measured none. Your server logs hold the visit count: check a request against the example string and the published IP list, which held 18 IPv4 ranges as read at 15:04 UTC that day (a count from the file, not of visits; the page does not call it complete). The page is silent on scripts; our JavaScript page covers them.

Should a site owner allow or block GPTBot?

Allowing or blocking GPTBot is your call; according to OpenAI's page, a Disallow "indicates" your content should not be used in training, and nothing there ties it to whether ChatGPT's search shows your website.

Search appearance is OAI-SearchBot's setting; the page notes search results can take "~24 hours" to reflect a robots.txt update. Where a site allows both OAI-SearchBot and GPTBot, OpenAI "may use the results from just one crawl for both use cases". Our search-index page reads ChatGPT's search side from that documentation.

A line is a request, not a lock. The robots.txt standard, RFC 9309, states its rules "are not a form of access authorization". For real access control, the RFC points to measures such as HTTP authentication. Our pages cover what a robots.txt line controls and what it cannot and the layers a block can sit at. OpenAI's page covers only its own tokens; Google's training token needs its own line.

What does our own robots.txt say about GPTBot?

The liamvi.com robots.txt names no AI crawler (read 11 November 2026), so GPTBot falls under the group every crawler shares. Rule lines exactly as served; one Sitemap line not shown:

``` User-agent: * Allow: / Disallow: /deck/ Disallow: /dashboard/ Disallow: /login/ Disallow: /api/ ```

Under RFC 9309, a crawler with no group naming it obeys the `User-agent: *` group, so any protocol-following crawler may fetch everything on liamvi.com except those four paths. Adding `User-agent: GPTBot` and `Disallow: /` would, by OpenAI's page, indicate our content should not be used in training, and would not change OAI-SearchBot.

Your own check takes a minute: open yoursite.com/robots.txt, look for a group naming GPTBot or OAI-SearchBot, and read it against the table. Which line your website needs is your decision. If OpenAI's page reads differently from what we wrote, tell us, and we will correct it in public.

What people actually ask us

Is GPTBot the same as ChatGPT search?

No. OpenAI's crawler page gives ChatGPT's search to OAI-SearchBot and calls each setting independent: a site can allow OAI-SearchBot while disallowing the training crawler.

Does blocking GPTBot remove data OpenAI already collected?

OpenAI's crawler page does not say whether already-collected data is removed. As read on 11 November 2026, disallowing GPTBot "indicates a site's content should not be used in training".

Sources

See where you stand today.