Will Scaling Improve Social Simulation with LLMs?

2026-07-02

AI Summary

The fidelity of large language models (LLMs) in social simulation and their response to computational scaling remain unclear. This study constructs 85 models based on the Qwen3 architecture (spanning 10¹⁸–10²⁰ FLOPs) and integrates 35 open-source LLMs to systematically introduce scaling laws into social simulation for the first time. Through pretraining scaling experiments, downstream task evaluation, and cognitive bias calibration, we investigate how computational scale affects opinion modeling, behavioral simulation, and longitudinal forecasting. Results show that performance on most tasks improves significantly with model scale—particularly among groups well-represented in English-language data—yet longitudinal prediction, minority-group viewpoints, and certain cognitive biases exhibit limited sensitivity to scaling. These findings reveal a misalignment between simulation fidelity and general-purpose capabilities, delineating the effective boundaries of scaling in social simulation.

Abstract

Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely. In this work, we investigate whether the current scaling paradigm in language modeling is likely to close these gaps, or whether simulation fidelity is orthogonal to general capabilities and therefore deserving of more research attention. We use scaling laws to study the relationship between LLMs' compute scale, general capability benchmarks, and the fidelity of social simulation in three representative sub-domains: opinion modeling, behavioral simulation, and longitudinal forecasting. Surprisingly, we discover strong compute scaling in all three settings, using a suite of 85 transformer LLMs with the Qwen3 architecture pre-trained on the DCLM web text corpus under fixed-compute budgets from $10^{18}$ to $10^{20}$ FLOPs. Then we evaluate 35 larger and more capable open-weight models up to 70B parameters, allowing us to predict downstream accuracy from loss. This reveals that the majority of behavioral and opinion simulation tasks will rapidly improve with scale, particularly when they involve populations that are well-represented in English web corpora. Longitudinal forecasting and underrepresented opinions scale more slowly, especially when they are less correlated with general knowledge and reasoning benchmarks like MMLU. In behavior simulation, scaling fails to improve model calibration with human cognitive biases like risk aversion, as well as human heuristics like learning correlated rewards from related tasks. On these tasks, even fine-tuned models fail to noticeably scale up performance from 0.5B to 8B parameters. Taken together, we conclude that scale will improve social simulations in most settings, but outliers exist, and improvements will be less reliable in low-resource domains.

Problem

• social simulation
• scaling laws
• simulation fidelity
• large language models
• behavioral modeling

Innovation

• scaling laws
• social simulation
• LLM fidelity
• behavioral modeling
• compute scaling