ChatGPT citation tracking: one study's test of how close Google results and OpenAI's API come to what the app cites
A ChatGPT citation report can only be as complete as the route its answers were collected through, and an independent study published on 11 October 2026 measured two common routes against the ChatGPT app itself. Put together, Google's search results and OpenAI's API "recover only about one quarter of cited URLs", in the words of the paper's abstract.
The figure behind that phrase is 28.3% for the GPT-5.3 Instant setting and 26.2% for GPT-5.5 Instant: of the pages the app cited in at least one of 20 runs of a question, averaged over 80 questions, that is the share the stand-ins found between them. The count includes the supporting pages attached to a reference the app selected, even when those pages are not shown as separate links, so the figure is not a count of visible links alone. At the level of whole websites the share rises to about half, 50.3% and 48.7%. The researchers are at the National University of Singapore and Xi'an Jiaotong University.
The limits belong beside the number. This is a preprint, not yet peer reviewed. It rests on 80 questions, asked through the researchers' own signed-in sessions, on GPT-5.x settings recorded between March and July 2026, before OpenAI put GPT-6 into ChatGPT on 7 October. Nothing here is a claim about GPT-6. The stand-ins were the authors' own builds, and no tracking product was tested.
We build software that logs the pages AI assistants cite in their answers, so what this study asks applies to our own data too, and we have not measured how far our own capture sits from what the app cites.
What does citation tracking mean?
Citation tracking means recording which pages an AI assistant links as sources when it answers a fixed set of prompts. Those prompts are then run again on a schedule, so that a change shows up over time. A citation is a linked source inside the answer; a mention is a brand named in the answer with no link, and AirOps' guide, for one, tells readers to track the two separately. The study read on this page counts citations more widely than the links a reader sees, a difference explained below and carried with every figure. The phrase also has an older academic sense, keeping count of who cites a research paper, which this page does not cover. Whatever a tracker reports, it first has to obtain each answer by some route: the app itself, OpenAI's API, or a search engine's results used as a guide to what the app might find. The study calls the last two proxies. We call them stand-ins here, because that is the job they do.
How did the researchers test the stand-ins?
The researchers asked the ChatGPT app the same 80 questions 20 times each and compared its citations with what Google and the API returned. For each query, all three were collected inside the same three-day window.
The queries come from Search Arena, a public collection of real requests people have put to search-enabled AI: 40 in English and 40 in Chinese, split evenly across four kinds of request, namely factual lookups, information synthesis, analysis and recommendations.
The app side was recorded in signed-in web sessions, from the stream of data the app sends to the browser while it answers. That stream shows the searches the app ran, the pages it retrieved and the references it selected. The main comparison covers GPT-5.3 Instant and GPT-5.5 Instant (labelled chatgpt-5.3-instant and chatgpt-5.5-instant in the paper), each with all 80 queries and 20 runs per query: 1,600 runs per setting, between 29 March and 29 July 2026. GPT-5.2 Instant and GPT-5.4 Thinking were deprecated before all their runs could be collected and play a smaller part in the paper.
The first stand-in is OpenAI's Responses API, the developer route, with web search switched on and no system prompt. The authors called the comparable models under their API names, gpt-5.3-chat-latest and gpt-5.5, and set reasoning effort to none for gpt-5.5 to bring it closer to the app's Instant setting: 20 calls per query, 1,600 calls per model. The second is Google: five pages of organic results per query, collected through the SerpAPI service, which came to an average of 43.2 results each.
OpenAI's own documentation explains why the API offers more than one list to compare. Its web search guide says a response carries inline citations, and that a separate sources field "returns the complete list of URLs the model consulted when forming its response". It adds that "the number of sources is often greater than the number of citations". The study used both lists: every source the API looked at, and each source it chose to cite.
What does the study count as a cited page?
One citation in any of the 20 runs of a query is enough for the study to count a page as cited. Every share is worked out query by query, then averaged across the 80.
Pages are matched at three levels. An exact URL is compared after small clean-ups: a leading www., tracking tags such as utm_, fragments and trailing slashes are removed. A website is its hostname, with subdomains kept apart, so a blog on its own subdomain is a different site from the main domain. An entity is a named thing, such as a brand, a product or an organisation, drawn from a page's title and snippet.
One caveat travels with every figure below. The study also counts as cited the supporting pages the app attaches to a reference it selected, even when those pages are not shown to the reader as separate links. The authors say so directly: their findings concern "the broader reference-selection process rather than only user-visible citations". A tracker that reads only the links a user can click is counting a narrower set, with each visible link as one source.
How much of what ChatGPT cites can the stand-ins see?
Google's results and OpenAI's API together held about a quarter of the exact pages the ChatGPT app cited in the study, and about half of its websites.
Across 80 questions and 20 runs each, Google's results and the pages OpenAI's API consulted together contained 28.3% of the exact pages GPT-5.3 Instant cited, and 26.2% of those GPT-5.5 Instant cited. At the website level the same pair contained 50.3% and 48.7% of the cited hostnames, and at the entity level 50.7% and 45.1%. Each share is of the pages, sites or entities the app cited in at least one of 20 runs, averaged over the 80 queries.
Alone, each stand-in saw less. At the exact-URL level, five pages of search results held 9.4% of the pages GPT-5.3 Instant cited and 6.4% of those GPT-5.5 Instant cited. The pages the API consulted held 22.9% and 22.5%.
That sets a ceiling no clever analysis can lift. Even a perfect ranking of everything the stand-ins found "can recover only 26.2%–28.3% of all cited URLs", in the paper's words, because a source that never appears in either list cannot be ranked at all.
It is worth being exact about what that means. The stand-ins' lists are not a smaller copy of the app's citations; they overlap with them only in part. Of the exact pages in the stand-ins' combined list for a query, 5.9% for GPT-5.3 Instant and 5.7% for GPT-5.5 Instant were among the app's citations in any of its 20 runs, averaged over the 80 queries, which is what the paper reports as the combined list's precision. So in the study's runs the stand-ins missed most of the pages the app cited, and most of the pages in their lists were never among the app's citations.
What does the gap do to citation rate and share of voice?
A citation rate or a share of voice built on stand-in data inherits the coverage gap the study measured. The paper frames the stakes in exactly these terms: understanding how pages are selected for citation, its introduction says, "can help brands improve their visibility and share of voice in generative-engine responses".
Our reading of what follows, and it is a reading rather than something the study measured: a citation rate is the share of tracked answers that link to your site, and a share of voice divides your citations or mentions by everyone's. Both are counted from whatever answers and sources the tracker collected. If the collected data holds about a quarter of the exact pages the app would cite, your page can be missing from it while the app is citing it, and the study does not say whether that happens to some kinds of page more than others. Nobody can correct for the gap from this paper alone.
The website level is the steadier part. An AI visibility report that also shows which of your sites and domains appear is reading the half the stand-ins matched better. Mentions sit outside this result: the study measured citations, not brands named without a link, so it says nothing about a mention rate. And it measured coverage, not any platform's citation rate or share of voice.
Why does the API come closer than Google's results?
We think OpenAI's API comes closer to the app's citations because of how each stand-in searches, a reading the study's figures fit but did not test.
In none of the GPT-5.3 Instant or GPT-5.5 Instant runs where the searches could be seen did the app search the exact wording of the question. The searches were visible in 1,339 of the 1,600 runs for GPT-5.3 Instant and in all 1,600 for GPT-5.5 Instant. That fits OpenAI's help text: ChatGPT search, it says, "typically rewrites your query" into more targeted queries when it passes a request to other search providers; our page on where ChatGPT's search results come from reads that help text in full. A stand-in that runs your wording through a search engine is searching for something the app never typed. The API's model, by contrast, runs searches of its own, and OpenAI's web search guide says the record of each one "will usually (but not always)" show what it searched for.
The API's citations make a narrower check, and a sharper one. Of the pages the API itself chose to cite, 16.4% were also cited by the app for GPT-5.3 Instant and 23.3% for GPT-5.5 Instant. At the website level those figures were 27.7% and 40.5%; the authors report both rises, from GPT-5.3 to GPT-5.5, as statistically significant. The search results were "generally weaker as a standalone proxy", in the authors' words, although they added coverage when combined with the API.
Why do exact pages match less often than whole websites?
Whole websites are easier for a stand-in to match than exact pages, and the app's run-to-run changes in the study help explain why. In the study's runs, the app kept returning to the same websites while the exact pages it looked at changed from one run to the next.
Of the distinct pages GPT-5.5 Instant looked at for a query across its 20 runs, 44.5% appeared in only one run; for GPT-5.3 Instant it was 40.8%. Only 7.7% and 9.4% appeared in at least 14 of the 20 runs. Comparing any pair of runs of the same query with a similarity score from 0 to 1, the overlap was about 0.3 for exact pages and about 0.4 for websites and entities.
These are pages the app looked at, not pages it picked as a source, and the difference matters. Pages that turned up in more runs were also more likely to be cited, a pattern the authors call observational: it does not show that appearing more often would earn a citation. What it helps explain is why a stand-in that finds the right website but a different page from it scores a miss at the URL level and a hit at the hostname level. It is also why the same question asked twice can come back with different sources, and why a single answer to one prompt is a thin basis for any report.
Bulk collection has one more trap. Two of the researchers' sessions switched silently to a fallback model, after 14 and 16 successful runs on the intended one. The authors' advice is that GEO monitoring "should therefore verify the serving model for each response", so that a tracker does not measure a fallback model without knowing it.
What did the study not test?
Three things sit outside the study: any tracking product, the model the app runs on today, and the route OpenAI itself ties to search in ChatGPT.
No vendor's tool was measured. The stand-ins were the researchers' own calls to Google and to the API, so the figures describe those builds, not any product sold for citation tracking.
The settings predate GPT-6 in ChatGPT. OpenAI's release notes date GPT-6 in the app to 7 October 2026, after the study's last run. The authors themselves warn that signals measured under one setting "may not transfer unchanged to another".
The API route tested is one of two. OpenAI's documentation says its Chat Completions API lets developers "directly access the fine-tuned models and tool used by Search in ChatGPT". The study used the Responses API instead, so it does not tell us how close that second route comes.
The sample is small and particular. Eighty queries, half of them in Chinese, drawn from a public dataset, are not the prompts your buyers type in your category. The sessions were the researchers' own and signed in, which is not how every tracker collects. And the work is observational: in the authors' words, it "does not directly evaluate GEO strategies", so it says nothing about what would raise your citations or improve your content.
A second preprint from the same week, by Mahe Chen and Nick Koudas, asks what drives AI citations; none of its findings are drawn on here.
Does this apply to Perplexity, Gemini or Google's AI Overviews?
The study measured one assistant alone, so its figures say nothing about Perplexity, Gemini or AI Overviews. Its authors write: "Future work will extend this study to more generative engines and model settings."
AI search tracking platforms usually report several engines side by side. Targetlytics' page, for one, lists "ChatGPT, Google AI Mode, Google AI Overviews, Claude, Gemini, Perplexity, Copilot, Grok, DeepSeek, Llama, Mistral and GLM" as the assistants it covers, and Surfer's comparison, read below, spans ChatGPT, Perplexity, Gemini, AI Mode and AI Overviews. A combined AI search figure therefore mixes engines whose stand-in gap has not been measured the way this study measured one assistant's, at least in the comparisons we found. Each AI search engine has its own app, its own developer route and its own way of showing a source, so a quarter for one engine says nothing about the share for Perplexity or Gemini. If your report blends engines, ask the questions below engine by engine.
Has anyone else compared the app with the API?
Surfer and Conductor, two SEO tool vendors, have published on the app-versus-API question, from opposite sides. This study is the only app-versus-API comparison we found on 19 October 2026 by academic researchers rather than a tracking vendor.
Surfer's comparison, last updated 4 October 2026, says it ran 1,000 prompts across five AI products twice, once through the interfaces people use and once through an API, for 13,779 answers. Surfer says on the same page that it "runs on scraped data", and it concludes: "API and the product share the model, but they're different systems."
Conductor's guide, last updated 12 January 2026, describes its own approach as API-based and argues that API monitoring "reflects how AI platforms and users actually operate". The page sets out that argument without a measurement of its own.
The study tested neither company's product. Each company describes its own method on the same page as its argument, so read both pages as positions and the study as one outside measurement of part of the question.
What do the tracking guides say about how answers are collected?
The guides that rank in search for "ChatGPT citation tracking" mostly describe what to track. Two say, briefly, that the route the answers come through matters.
Winston Digital Marketing's guide, updated 30 September 2026, puts it this way: "How the answers are captured. Some tools read answers from the consumer apps people actually use. Others call the model APIs, which can return different sources for the same question. Ask which method each engine uses, because it changes what the numbers mean."
OutreachBloom's guide, updated 15 October 2026, asks whether ChatGPT citations can be tracked through OpenAI's API and answers "Partly." It adds: "No API exposes what ChatGPT told real users, so every tracker samples answers, including a script you write yourself."
AirOps, in a guide dated 10 September 2026, sets out four methods: testing prompts by hand, reading referral traffic in GA4, checking server logs for OpenAI's crawlers, and using an AI citation tracking platform. It advises testing "at least three different phrasings of the same question". Targetlytics says of its own source list: "Some assistants don't show all their sources, so the list is a strong sample, not a complete one."
As of 19 October, none of these guides, nor the other readable ranking pages listed under Sources, cites the study, which now puts a measured size on the gap those sentences describe, for both stand-ins and one period.
What should you ask a ChatGPT citation tracker?
Ask any ChatGPT citation tracker how it collects each answer and what unit it counts a citation in. Get both answers before you trust a page-level number. These are our questions for SEO and content teams reading these reports, shaped by what we build; apply them to us too.
- Where does each answer come from: the app, signed in or not, an API, or search results? The study found the route decides how much of the app's citation data you can see at all.
- Is a citation counted for an exact page, a website, or both, and can you see the website level? The stand-ins matched about half of the cited websites but about a quarter of the exact pages.
- How many runs sit behind each prompt, and is the figure a share of runs or a "cited at least once" count? The study's shares use the second, and they would read differently as the first.
- Does it record the searches the app actually ran, or only the wording you typed? The app searched the exact wording in none of the observed GPT-5.3 and GPT-5.5 Instant runs.
- Does it record which model answered each run? A silent fallback can sit inside a trend line unnoticed.
- What happens to your history when OpenAI changes the model? Two of the study's four settings were deprecated during collection, and GPT-6 has since arrived in the app.
What AI visibility vendors write down about their own numbers, from formulas to prompt sources, is the subject of our read of their documentation.
What does this change about reading your own report?
Page-level citations in a tracking report are best read as a sample, and website-level presence as the steadier signal.
A page that is absent from a report built only on a stand-in may still be cited in the app, and a page that appears in one may not be. A page cited once, in one run, is weak evidence on its own, because in the study's runs the app looked at different pages from one run to the next. A sudden change in your numbers after a model release may say as much about the instrument as about your content. None of that tells you how to earn a citation, and nothing here promises one; how we read any citation index, including our own applies the same caution to our measurements.
Which tracker you use, and how far you trust its number, is your call. The study gives you better questions to ask, not the answer.
Sources
- Jiang, Zhang, Xing, Chang and Xiao, Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations, arXiv 2610.15382, v1, 11 October 2026: https://arxiv.org/abs/2610.15382 (full text: https://arxiv.org/html/2610.15382v1) — checked 19 October 2026
- OpenAI, web search tool guide (API documentation): https://developers.openai.com/api/docs/guides/tools-web-search — checked 19 October 2026
- OpenAI Help Center, "Searching the web with ChatGPT": https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt — checked 19 October 2026
- OpenAI, ChatGPT release notes (7 October 2026, GPT-6 in ChatGPT): https://help.openai.com/en/articles/6825453-chatgpt-release-notes — checked 19 October 2026
- Chen and Koudas, What Drives AI Citations? Content, Retrieval and Model Knowledge in Generative Engine Optimization (GEO), arXiv 2610.15033, 10 October 2026: https://arxiv.org/abs/2610.15033 — checked 19 October 2026
- Surfer, "Why You Cannot Track AI Visibility Through an API" (last updated 4 October 2026): https://surferseo.com/blog/llm-scraped-ai-answers-vs-api-results/ — checked 19 October 2026
- Conductor, "Scraping vs. API: How to Track Brand Visibility in AI Search" (last updated 12 January 2026): https://www.conductor.com/academy/scraping-vs-api/ — checked 19 October 2026
- Winston Digital Marketing, AI citation tracking guide (updated 30 September 2026): https://www.winstondigitalmarketing.com/playbooks/best-ai-citation-tracking-tools/ — checked 19 October 2026
- OutreachBloom, "How to Track Your Brand's ChatGPT Citations" (updated 15 October 2026): https://outreachbloom.com/how-to-track-your-brands-chatgpt-citations — checked 19 October 2026
- AirOps, "ChatGPT Citation Tracking: Four Methods That Show If You're Being Cited" (10 September 2026): https://www.airops.com/blog/chatgpt-citation-tracking — checked 19 October 2026
- Targetlytics, citation tracking product page: https://targetlytics.com/en/features/citation-tracking — checked 19 October 2026
- The pages ranking for this question that we read on 19 October 2026: seoClarity, AirOps, Targetlytics, OutreachBloom, Winston Digital Marketing, Finseo, Georion and TrackMyVisibility, plus a Reddit r/bigseo thread that returned no readable text.
