作者: Dmitrij ŻatuchinarXiv 2610.00253

System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer(大语言模型品牌推荐中的系统归因:单条回复可识别系统,聚合品牌画像无法跨域迁移)于 2026-09-23 提交到 arXiv,作者 Dmitrij Żatuchin。去 arXiv 读原文,或下载 PDF。

为什么和被引用有关

最重要的一点。 通过聚合品牌推荐数据来评估 AI 可见性,并不能反映模型的稳定属性。模型的真正指纹体现在表面排版结构中,而品牌推荐结果会随任务类型和采集环境剧烈变化。

摘要

Audits of AI visibility summarise the brand recommendations of deployed language models into per-system profiles. We test whether such a profile describes the system on one corpus of 6,475 stored responses (6,324 analysable) collected between December 2025 and February 2026 from five deployed endpoints across gift-recommendation, corporate-reputation and category-ownership queries. The collection harness cut many answers short: 83.1% of Gemini 3 Flash answers in category ownership end mid-sentence under a 1,024-token output cap. With every answer cut to its first 800 characters, a character n-gram classifier cross-validated by prompt attributes one response to GPT-5.2, Gemini 3 Flash, Gemini 3 Flash with search, Grok or Perplexity sonar-pro with 97.84% accuracy (5,028 responses, 383 prompts, majority class 31.5%, 30 split seeds). Length alone falls to the majority rate, 24 formatting statistics reach 95.79%, and masking brand names and capitalised tokens leaves 97.72%. Held-out query conditions keep 97.43% weighted by size and 88.0% unweighted; in a retrieval-grounded arm that changes the harness, no Grok answer is attributed to Grok (0/120). Aggregated into 50 model-by-domain-by-condition units, twelve behavioural features separate four systems at 66.53% under grouped cross-validation, against a label-permutation null with mean 33.71% and 95th percentile 46.0%. Across domains the aggregate profile fails: a forest trained on category-ownership units assigns all 22 gift units to the wrong system, consistent with a reversal in brand volume (8.41 against 0.94 brands per response in gifts, 3.01 against 3.91 in category ownership), while single responses transfer at 89.92% balanced accuracy. The surface form of an answer carries the system across the query domains tested; aggregated brand behaviour does not, and the uncrossed design cannot separate the system from the domain or the harness.

译文:AI 可见性审计通常将部署的语言模型的品牌推荐总结为各个系统的专属画像。我们在一个包含 6475 条存储回复(6324 条可分析)的语料库上测试了此类画像是否能够准确描述系统。该语料库于 2025 年 12 月至 2026 年 2 月期间从五个已部署的终端收集,涵盖礼物推荐、企业声誉和品类归属等查询。数据采集工具截断了许多回答:在 1024 个 token 的输出上限下,品类归属查询中 83.1% 的 Gemini 3 Flash 回答在句子中途中断。当每条回答截断至前 800 个字符时,基于提示词交叉验证的字符 n-gram 分类器能够以 97.84% 的准确率将单条回答归因于 GPT-5.2、Gemini 3 Flash、带搜索的 Gemini 3 Flash、Grok 或 Perplexity sonar-pro(共 5028 条回答,383 个提示词,多数类占比 31.5%,30 个划分随机种子)。单纯依靠文本长度分类准确率下降至多数类基准,24 个格式统计特征达到 95.79% 的准确率,屏蔽品牌名称和大写词元后仍保留 97.72% 的准确率。在相同采集工具下的保留查询条件下,样本量加权准确率维持在 97.43%,未加权为 88.0%;在改变采集环境的检索增强分支中,没有任何 Grok 回答被归因于 Grok(0/120)。当聚合为 50 个模型-领域-条件单元时,基于分组交叉验证,12 个行为特征以 66.53% 的准确率区分了四个系统,而标签置换的零分布均值为 33.71%,第 95 百分位数为 46.0%。跨领域测试时,聚合画像彻底失效:在品类归属单元上训练的随机森林模型将全部 22 个礼物单元归入错误的系统,这与品牌提及数量的逆转一致(礼物场景中每条回答提及 8.41 个对比 0.94 个品牌,品类归属场景中为 3.01 个对比 3.91 个),而单条回答迁移的平衡准确率为 89.92%。回答的表面格式特征在所测试的查询领域中承载了系统标识;聚合的品牌行为特征则无法做到,未正交的实验设计无法将系统与查询领域或采集工具分离开来。

关键术语

论文原文

作者指出了生成式引擎优化中针对不同模型进行独立衡量的基础假设。

Generative engine optimisation treats those answers as a surface that content can be shaped for and measured on Aggarwal et al. (2024) . This paper starts from that premise and tests the assumption beneath any per-system measurement of it: that an answer can be attributed to the system that produced it, and that what the measurement observes is a property of that system.

译文:生成式引擎优化将这些回答视为可以进行内容定制和效果衡量的载体。本文从这一前提出发,检验了所有针对具体系统评估背后的基础假设:即一条回答能否归因于生成它的系统,以及测量所观察到的现象是否属于该系统本身的属性。

聚合行为特征虽然在单一场景内能够区分系统,但迁移到新任务时会完全失效。

Twelve behavioural features of brand volume, concentration, sentiment and source behaviour separate four systems at 66.53% inside the collection window, and a classifier moved to a different query domain assigns every gift-domain unit to the wrong system.

译文:包含品牌数量、集中度、情感倾向和信源行为在内的十二个行为特征,在采集周期内能以 66.53% 的准确率区分四个系统,但将分类器迁移到不同查询领域时,却把所有礼物领域的样本单元判定为错误系统。

研究发现输出上限导致了系统性的回答丢失,而非随机丢失。

Empty responses. 136 stored GPT-5.2 responses in the category-ownership subset have zero-length text and no error flag. They are a behaviour of that endpoint under the 1,024-token completion cap of that collection, and they are missing not at random.

译文:空回复。在品类归属子集中,存储的 136 条 GPT-5.2 回复文本长度为零且没有报错标识。这是该接口在本次采集的 1024 词元上限下的输出表现,这些数据并非随机缺失。

论文里的图

该图展示了聚合行为画像跨任务迁移完全失效,而单条回答的表层特征模型依然保持高准确率的鲜明对比。

Figure 5: The same cross-domain question at two units of observation, Gemini 3 Flash against GPT-5.2, chance 50%. Left,

图来自论文 arXiv:2610.00253,版权归作者。原图说明:Figure 5: The same cross-domain question at two units of observation, Gemini 3 Flash against GPT-5.2, chance 50%. Left, aggregated behavioural profiles: units correct out of units tested for each of the four train and test pairs, one forest fit each. Right, single responses on the gift and category-ownership pairs, on the stored text, on 800-character prefixes and on masked prefixes.

作者自己说的含义

可以怎么做

  1. 针对不同的业务品类独立测试 AI 搜索表现,停止依赖跨领域的全局品牌可见性报表。 依据:品类归属数据训练的模型在测试 22 个礼物领域样本时全部判断错误,品牌提及数量在跨场景时发生反转。
  2. 在统计 AI 可见性指标前,严格检查监控工具是否存在接口截断或输出词元限制。 依据:在 1024 个词元上限设置下,Gemini 3 Flash 在品类归属查询中有 83.1% 的回答未说完就遭到截断。
  3. 排查提示词测试中的排版结构和标记符号,确保分析结论来自真实推荐内容。 依据:单纯利用 24 个排版格式统计指标,模型识别准确率就达到了 95.79%。

注意。 本研究仅基于三个月内在礼物推荐、企业声誉和品类归属三个领域收集的 6324 条可用回复进行测试,未涵盖其他非商业场景或更新版本的模型终端。

一句话

许多团队试图通过聚合数据描摹各个 AI 引擎对品牌的偏好倾向。本研究表明这种聚合画像在不同业务场景和抓取工具下极不稳定。准确评估生成式引擎的表现,需要严密控制业务领域差异和数据采集环节的系统偏差。

去 arXiv 读论文 →

这篇解读根据论文原文写成,发布前有人看过。引用的段落是原话,数字都是作者自己报告的。方法和完整结果请读论文。