Social Simulations with Large Language Model Risk Utopian Illusion

2025-10-24

AI Summary

This study identifies a pervasive “utopian bias” in large language models (LLMs) when simulating human social behavior—namely, an over-idealization of actions that diverges from the complexity and heterogeneity of real-world social interaction. This bias, previously unexamined systematically, undermines the validity of LLMs in sociocognitive applications. Method: We propose the first multidimensional analytical framework for assessing social-cognitive biases in LLMs, grounded in multi-agent chatroom simulations. Our framework quantifies five linguistic dimensions: social role bias, primacy effect, positivity bias, social desirability bias, and behavioral consistency. Contribution/Results: Evaluating eight state-of-the-art LLMs across three architectural families, we demonstrate that their behavioral representations are strongly driven by social desirability bias, leading to statistically significant deviations from empirically observed human interaction patterns. This work establishes both theoretical foundations and methodological tools for risk assessment and trustworthy modeling of LLMs in social contexts.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reliable simulation of human behavior is essential for explaining, predicting, and intervening in our society. Recent advances in large language models (LLMs) have shown promise in emulating human behaviors, interactions, and decision-making, offering a powerful new lens for social science studies. However, the extent to which LLMs diverge from authentic human behavior in social contexts remains underexplored, posing risks of misinterpretation in scientific studies and unintended consequences in real-world applications. Here, we introduce a systematic framework for analyzing LLMs' behavior in social simulation. Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions, providing a simple yet effective method to examine emergent social cognitive biases. We conduct extensive experiments involving eight representative LLMs across three families. Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it, shaped by the social desirability bias. In particular, LLMs show social role bias, primacy effect, and positivity bias, resulting in "Utopian" societies that lack the complexity and variability of real human interactions. These findings call for more socially grounded LLMs that capture the diversity of human social behavior.

Problem

• Analyzing divergence between LLM-simulated and authentic human social behavior
• Identifying emergent cognitive biases in multi-agent language model interactions
• Addressing risks of utopian illusions in social science simulations

Innovation

• Systematic framework analyzes LLM social behavior
• Simulates multi-agent interactions via chatroom conversations
• Examines five linguistic dimensions for cognitive biases