LLM-Based Social Simulations Require a Boundary

2025-06-24

AI Summary

LLM-driven social simulation often suffers from behavioral homogenization—producing “average-persona” outputs—thereby failing to capture real-world social heterogeneity and undermining its reliability for social science research. Method: We systematically delineate the applicability boundaries of LLM-based social simulation by proposing three core evaluation criteria—alignment, consistency, and robustness—and designing an operational assessment checklist. Integrating LLM-powered virtual agents with classical agent-based modeling, we construct a heuristic validation framework that preserves macro-level pattern discovery while explicitly identifying trustworthy application scenarios. Contribution/Results: This work innovatively transforms boundary specification from abstract discourse into a structured, empirically testable standard system. It significantly enhances the interpretability and empirical validity of simulation outcomes, providing a rigorous methodological foundation for leveraging LLMs in social science research.

Technology Category

• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Application Domains: Humanities & Computational Social Science

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This position paper argues that large language model (LLM)-based social simulations should establish clear boundaries to meaningfully contribute to social science research. While LLMs offer promising capabilities for modeling human-like agents compared to traditional agent-based modeling, they face fundamental limitations that constrain their reliability for social pattern discovery. The core issue lies in LLMs' tendency towards an ``average persona'' that lacks sufficient behavioral heterogeneity, a critical requirement for simulating complex social dynamics. We examine three key boundary problems: alignment (simulated behaviors matching real-world patterns), consistency (maintaining coherent agent behavior over time), and robustness (reproducibility under varying conditions). We propose heuristic boundaries for determining when LLM-based simulations can reliably advance social science understanding. We believe that these simulations are more valuable when focusing on (1) collective patterns rather than individual trajectories, (2) agent behaviors aligning with real population averages despite limited variance, and (3) proper validation methods available for testing simulation robustness. We provide a practical checklist to guide researchers in determining the appropriate scope and claims for LLM-based social simulations.

Problem

• Establish boundaries for LLM-based social simulations' reliability
• Address alignment, consistency, robustness in simulated social behaviors
• Guide proper validation methods for simulation robustness

Innovation

• Establish boundaries for LLM-based social simulations
• Address alignment, consistency, and robustness issues
• Focus on collective patterns and proper validation