Exploring Large Language Model Agents for Piloting Social Experiments

2025-08-12

AI Summary

Low intelligence of computational agents in current social experiments constrains their applicability and ecological validity. Method: We propose the first large language model (LLM)-based intelligent agent framework designed specifically for social experiment pre-enactment. It integrates social science theory to construct high-fidelity interactive environments and supports intervention design alongside automated acquisition and analysis of multimodal data—including behavioral logs, survey responses, and dialogue transcripts. LLMs serve as the cognitive core for individual decision-making, systematically embedded within experimental paradigms and data engineering pipelines. Contribution/Results: Evaluated across three canonical social experiments, the framework faithfully reproduces both quantitative outcomes (e.g., obedience and cooperation rates) and qualitative patterns (e.g., discourse strategies and group dynamics), demonstrating strong validity and scalability. This work overcomes the intelligence bottleneck of traditional agent-based modeling, establishing an interpretable, iterative “silicon-based pre-enactment” paradigm for social science research.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Computational social experiments, which typically employ agent-based modeling to create testbeds for piloting social experiments, not only provide a computational solution to the major challenges faced by traditional experimental methods, but have also gained widespread attention in various research fields. Despite their significance, their broader impact is largely limited by the underdeveloped intelligence of their core component, i.e., agents. To address this limitation, we develop a framework grounded in well-established social science theories and practices, consisting of three key elements: (i) large language model (LLM)-driven experimental agents, serving as "silicon participants", (ii) methods for implementing various interventions or treatments, and (iii) tools for collecting behavioral, survey, and interview data. We evaluate its effectiveness by replicating three representative experiments, with results demonstrating strong alignment, both quantitatively and qualitatively, with real-world evidence. This work provides the first framework for designing LLM-driven agents to pilot social experiments, underscoring the transformative potential of LLMs and their agents in computational social science

Problem

• Enhancing agent intelligence in computational social experiments
• Developing LLM-driven framework for social experiment simulations
• Validating framework with real-world experiment replications

Innovation

• LLM-driven agents as silicon participants
• Methods for implementing diverse interventions
• Tools for multi-modal data collection