‘If I Had to Buy Just ONE: Galaxy S26 Ultra’: Auditing AI-Generated Product Recommendations was submitted to arXiv on 2026-09-16 by Lucas G. Uberti-Bona Marin, Thales Bertaglia, Giovanni Astante, Bram Rijsbosch, Gijs van Dijck, Anikó Hannák and others. Read it on arXiv or as a PDF.
Why it matters for getting cited
- Chatbots frame recommendations differently, with ChatGPT expressing a first-person product preference in 79% of product-recommending responses, compared with 7% for Gemini and 2% for Google AI Overviews.
- Engines rarely cite the same websites, sharing only 5.4% of domains on average between ChatGPT and Gemini interfaces, with no domain in common across 76.7% of comparisons.
- Developer APIs do not match web interfaces, showing a mean domain overlap of only 12.0% for ChatGPT and 14.8% for Gemini on identical queries.
- Repeated user queries alter the recommendations and surface different citation sources, showing that single outputs do not represent stable platform behavior.
The main takeaway. Auditing or optimizing for AI recommendations requires monitoring repeated user-facing queries on web interfaces because developer APIs return different citations and products.
The abstract, as written
Consumers increasingly use AI chatbots for advice on what to buy. With companies like OpenAI and Google monetising their AI through advertising, this raises difficult questions about the bias and impartiality of such advice. In response, we conduct an AI audit of popular chatbots using real commercial-advice queries. First, we curate a dataset of 2,528 real commercial-advice queries (ConsumerQ). Then, we evaluate 1,536 responses to product queries from popular AI chatbots: ChatGPT (chatbot and API), Google Gemini (chatbot and API), and Google Search (AI Overviews). We find that ChatGPT expresses a first-person product preference in 79% of product-recommending responses, compared with 7% for Gemini and 2% for AI Overviews, while the products recommended often change across repeated requests. Displayed sources vary strongly: for the same query, the ChatGPT and Gemini interfaces share only 5.4% of domains on average, with no domain in common in 76.7% of comparisons. APIs provide a different view from their corresponding interfaces, with mean domain overlaps of 12.0% for ChatGPT and 14.8% for Gemini, and also differ in the types and layers of source information they expose. Our findings show that neither isolated responses nor API observations can be assumed to represent the commercial advice consumers encounter. Independent audits of AI-mediated commercial advice should therefore account for repeated responses, consumer-facing conditions, and the source layer being observed.
Key terms
- Commercial-Advice Query: A search or chat prompt where a user explicitly asks an AI assistant for recommendations, comparisons, or validation regarding products or services to buy.
- First-Person Preference: A framing style where the AI assistant uses personal pronouns such as ‘I’ or ‘my pick’ to endorse a specific product over alternatives.
- Jaccard Overlap: A metric that divides the number of shared items by the total unique items across two sets to assess source or product similarity.
- AI Overviews: An automated generative search summary displayed directly within Google search results to answer user queries with synthesized web content.
- Grounding Sources: The specific web pages or reference documents retrieved by an AI system to anchor its generated response in factual information.
From the paper
AI chatbots diverge significantly in how directly they endorse specific consumer products.
ChatGPT expressed a first-person preference in 79% of product-recommending responses, compared with 7% for Gemini and 2% for AI Overviews, and labelled products as “best” considerably more often. The systems therefore differ not only in what they recommend, but in how they present the recommendation: from explicit personal endorsement to a more qualified presentation of alternatives.
Chatbots rarely rely on the same publisher domains when recommending products to consumers.
For the same query, ChatGPT and Gemini share, on average, 5.4% of displayed domains, and 76.7% of comparisons share no domain at all.
Audits conducted over developer APIs fail to reflect the sources shown to real users on web interfaces.
For identical queries submitted moments apart, ChatGPT’s interface and API share only 12.0% of displayed domains on average; for Gemini, the overlap is 14.8%.
Figures from the paper
ChatGPT uses first-person endorsements and labels single products as the best far more frequently than Gemini or Google AI Overviews.
Figure from the paper, arXiv:2609.18729, by its authors. Caption as written: Figure 2: ChatGPT more often highlights options as the best and/or its pick/recommendation. Product recommendation framing across the three consumer interfaces. Bars show the share of the answers naming a product that contains each measured property; whiskers are 95% bootstrap CIs over the queries.
When interfaces and APIs cite different sources for identical queries, the disagreement almost always stems from entirely different domains rather than different pages on the same site.
Figure from the paper, arXiv:2609.18729, by its authors. Caption as written: Figure 5: When the interface and API disagree on sources, they usually disagree on the domain itself. Pages from domains appearing on only one side account for 79.8% of the ChatGPT interface–API union and 82.5% of the Gemini union. The remaining pages are either exact matches or different pages from domains appearing on both sides.
What the authors say it means
- Brands cannot rely on developer APIs to track how their products are cited by commercial AI chatbots.
- Publishers must track citations across multiple search passes because individual sessions retrieve inconsistent sets of web domains.
- Consumer trust is shaped differently across tools due to ChatGPT using first-person endorsements while Google tools use neutral overviews.
What to do with it
- Audit AI engine visibility directly through consumer web interfaces rather than backend APIs. Supported by: The paper found that the mean domain overlap between web interfaces and developer APIs was only 12.0% for ChatGPT and 14.8% for Gemini.
- Sample brand queries across multiple repeated requests to account for recommendation variance. Supported by: The paper demonstrated that recommended products and cited sources regularly change across repeated submissions of the same prompt.
- Target a wide variety of review publications instead of expecting universal cross-engine citations. Supported by: The paper showed that ChatGPT and Gemini shared only 5.4% of cited domains on average, with 76.7% of comparisons having zero shared domains.
Caveat. The evaluation examined 1,536 responses across 117 English physical-product queries collected from a single European location and did not measure non-English queries, service purchases, or long-term algorithmic shifts.
In short
AI product recommendations are volatile and vary sharply between chatbot providers and repeat queries. Web interfaces present different citations than developer APIs, and ChatGPT frequently issues direct first-person endorsements. Marketers and researchers must monitor live user interfaces over multiple sessions rather than trusting static API data.
Written from the paper's own text and checked by a person before publishing. Quoted passages are verbatim; numbers are the authors' own. For the method and full results, read the paper.