TL;DR
- ChatGPT Web retrieves many pages and shows only a few as citations. GEO work that stops at “did we get retrieved” measures the wrong step. Our paper traces what happens in between, across four model settings.
- Instant models mostly run one search round. GPT-5.4 Thinking ran 3.2 rounds and 11.8 executed queries per answer, and 71% of its answers used more than one round. No setting kept the user’s wording: exact matches were 0% to 5.5%.
- Later rounds get narrower: queries with a site: restriction rose from 25% to 55%, quoted phrases from 1% to 23%. Chinese questions were often searched in English only.
A traditional search engine decides visibility in two steps, retrieval and ranking. A generative engine adds a third: after retrieving pages it selects which ones to cite. Most measurement work on GEO looks only at the final answer, which pages were cited and how systems differ. Our paper Understanding ChatGPT Citations looks at the steps before that, because a page can be retrieved every time and never be shown.
This is the first of four write-ups. This one covers how we observed the pipeline and what query rewriting looks like. The others cover which retrieved pages get cited, whether Google and the API can stand in for ChatGPT Web, and whether citations can be predicted from the outside.
How we observed ChatGPT Web
We collected the server-sent event traces that ChatGPT Web streams to the browser during an answer, from our own authenticated sessions, between 29 March and 29 July 2026. The trace exposes the queries the model actually executed, each retrieval round, the candidate pages it pulled, and the final citation list. We decomposed the pipeline into three stages for analysis: query rewriting, candidate retrieval, and citation selection.
The query set is 80 questions drawn from Search Arena, a public dataset of in-the-wild interactions with search-augmented models. They cover four intents, factual lookup, information synthesis, analysis and recommendation, with 20 questions per intent, half in Chinese and half in English.
| Setting | Queries | Runs per query | Total runs |
|---|---|---|---|
| ChatGPT 5.2 Instant | 32 | 7 to 28 | 547 |
| ChatGPT 5.3 Instant | 80 | 20 | 1,600 |
| ChatGPT 5.4 Thinking | 30 | 4 to 20 | 491 |
| ChatGPT 5.5 Instant | 80 | 20 | 1,600 |
The 5.2 and 5.4 collections are partial because those models were retired before we finished, and the 5.4 query set does not match the Instant sets. Comparisons across settings are therefore descriptive. Two sessions also switched silently to a fallback model partway through, which is itself a finding: anyone measuring ChatGPT has to verify which model served each response.
Finding 1: Instant answers are one search, Thinking answers are many
An Instant answer usually issued one search round with one or two executed queries. GPT-5.4 Thinking behaved differently: 11.82 executed queries and 3.20 retrieval rounds per answer on average, with 71.3% of answers using more than one round. Among Instant settings, 5.2 issued 2.36 queries per answer, 5.3 issued 1.06 and 5.5 issued 1.45.
The Thinking model’s multi-round behaviour varied by intent: 87.6% of information-synthesis answers and 85.2% of analysis answers used several rounds, against 10.8% of factual lookups. We report these as descriptive because the intent groups differ in size.
Finding 2: ChatGPT almost never searches your words
Across all four settings, the executed query exactly matched the user’s question in 5.5% of 5.2 runs and in 0% of the others. The model rewrites before it searches. For questions asked in Chinese, 5.2 issued both Chinese and English queries in 96.8% of runs, while 5.3 and 5.5 searched in English only in 67.0% and 69.6% of runs.
For anyone tracking visibility, this means the query that reaches the index is not the query the buyer typed. A page optimised for the buyer’s phrasing may never be matched against it.
Finding 3: later rounds get narrower, not broader
When a Thinking answer ran several rounds, the later queries were more specific. The share of executed queries carrying a site: restriction rose from 25.1% in the first round to 54.7% in later rounds. Quoted phrases rose from 1.0% to 22.7%. Of 4,077 queries after the first round, only 13 repeated an earlier query. The model narrows toward particular hosts and exact phrases as it goes. Whether that produces better candidates or better citations is not something this part can say; it describes the search behaviour only.
What this part does not show
The traces are observational and come from our own sessions, so they describe what ChatGPT Web did for these 80 questions in this window, not what it does in general. The 5.2 and 5.4 collections are incomplete, and executed queries were missing from 261 of the 1,600 runs for 5.3. Nothing here says that matching the rewritten queries would change what gets cited.
Continue with Part 2: among the pages it retrieves, which does it cite?