Silicon Sampling via Cross-Survey Transfer

2026-07-03

AI Summary

This study addresses a critical limitation in existing silicon-based sampling methods, which predominantly rely on distribution-level evaluations and thus fail to assess a model’s ability to predict individual-level responses. The authors propose the first individual-level cross-questionnaire transfer evaluation framework, leveraging large language models (LLMs) to predict respondents’ answers to unseen survey items based on their partial responses within the same questionnaire. Using data from the 2024 Taiwan Election and Democratization Study (TEDS), experiments demonstrate that zero-shot LLMs achieve 52% accuracy on unseen questions—approaching the performance of supervised models—and reveal a stable hierarchy of predictability across political attitude constructs, ranging from 23% to 67%. This framework not only uncovers a construct-level predictability hierarchy but also offers novel empirical insights into variance collapse and safety alignment effects.

Abstract

Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.

Problem

• silicon sampling
• cross-survey transfer
• individual-level prediction
• survey research
• large language models

Innovation

• cross-survey transfer
• silicon sampling
• individual-level prediction
• variance collapse
• safety alignment