Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

2026-08-29

AI Summary

研究测试了丰富的人物数据是否能提高大语言模型作为人类替身的表现，发现尽管模型在预测平均响应上表现良好，但在捕捉个体差异方面效果不佳。

Abstract

LLMs are increasingly used as human surrogates, often on the premise that richer persona data could make them substitutes or exploratory tools for specific individuals. We test this premise across four datasets covering more than 400,000 participants and more than 6,000 survey items and experimental outcomes. LLMs perform well at the aggregate level: their average responses closely align with average human responses to the same items. But this success largely reflects predicting each item's average human response. Once each item's human mean is removed, LLM predictions explain only 3.05% of the remaining respondent-specific variation, far below the 53.6% human test-retest benchmark. Richer personas, model variants, and fine-tuning do not close this gap. In variance analyses, once item means are removed, the reliable remaining signal is person-by-item. It captures how a respondent departs from the mean on a particular item and is about 8.9x larger than the stable person effect. Persona data encode the respondent, but not this item-specific deviation. LLM responses also compress human response distributions, using less spread, fewer response categories, and distorted distributional shapes. We call this pattern item-mean surrogacy. Current LLM surrogates can approximate item averages, but not the distributions or respondent-specific deviations needed to replace individual humans. We propose four empirical tests for LLM-based human-surrogate claims.

Problem

• LLMs
• human surrogates
• persona data
• response variation
• item-mean surrogacy

Innovation

• item-mean surrogacy
• persona data
• response distribution
• individual variation
• human surrogate