Psychometric Comparability of LLM-Based Digital Twins

2025-12-22

arXiv.org

AI Summary

This study systematically applies construct validity theory from psychometrics to evaluate large language model (LLM) digital twins, investigating their capacity to psychologically emulate human participants. Integrating methods from construct validation, personality modeling, word association network analysis, cross-lingual free-text generation, and behavioral bias simulation, the research examines LLMs’ psychological agency along two dimensions: construct representation and nomological networks. Findings indicate that LLM-based digital twins perform adequately at the aggregate level but exhibit significant limitations in item-level correlations, replication of heuristic biases, invariance of network structures, and individual-level expression across languages. These results delineate the alignment boundaries and contextual applicability of LLMs as proxies for human psychological processes.

Abstract

Large language models (LLMs) are used as"digital twins"to replace human respondents, yet their psychometric comparability to humans is uncertain. We propose a construct-validity framework spanning construct representation and the nomological net, benchmarking digital twins against human gold standards across models, tasks and testing how person-specific inputs shape performance. Across studies, digital twins achieved high population-level accuracy and strong within-participant profile correlations, alongside attenuated item-level correlations. In word association tests, LLM-based networks show small-world structure and theory-consistent communities similar to humans, yet diverge lexically and in local structure. In decision-making and contextualized tasks, digital twins under-reproduce heuristic biases, showing normative rationality, compressed variance and limited sensitivity to temporal information. Feature-rich digital twins improve Big Five Personality prediction, but their personality networks show only configural invariance and do not achieve metric invariance. In more applied free-text tasks, feature-rich digital twins better match human narratives, but linguistic differences persist. Together, these results indicate that feature-rich conditioning enhances validity but does not resolve systematic divergences in psychometric comparability. Future work should therefore prioritize delineating the effective boundaries of digital twins, establishing the precise contexts in which they function as reliable proxies for human cognition and behavior.

Problem

• psychometric comparability
• digital twins
• large language models
• construct validity
• nomothetic span

Innovation

• psychometric comparability
• digital twins
• construct validity
• large language models
• nomothetic span