Calibrating the Instrument: Controllability of an LLM-Driven Synthetic Population

2026-07-01

AI Summary

This study investigates whether large language model (LLM)-driven synthetic populations exhibit controllability—defined as the capacity to generate ordered, reproducible responses to external stimuli of known valence that align with a predefined population structure. In the fictional city of Montelago, we constructed 120 synthetic agents with explicit latent structures and evaluated their reactions to seven institutionally framed messages of varying valence using generative synthetic populations (GSP), LLM-based agents, a pre-registered validation framework (SIVE), and temperature parameter sweeps. All seven pre-registered metrics were satisfied across all temperature settings; a “weakly positive” message initially misclassified as negative was identified and corrected. Instrument noise remained stable at approximately half the magnitude of inter-agent variability, and individual-level trajectories revealed dynamic patterns obscured in aggregate statistics. The work establishes controllability as a core criterion for internal validity in synthetic populations and repurposes calibration failures into a diagnostic tool uncovering unexpected interactions between message sentiment and agent trust.

Abstract

Generative Synthetic Populations (GSP) -- the convergence of population synthesis, agent-based modelling, and LLM agents -- are attracting growing interest for urban simulation and institutional communication research. Before any GSP instrument is used on a real population, a more basic question must be answered: does it respond to stimuli of known valence in an ordered, replicable, group-structured way? We call this controllability. We ask not whether a synthetic population tracks humans, but whether it tracks itself: whether the latent structure we impose on it is recovered in its own responses. This internal-validity question is logically prior to any claim about external validity, just as characterising an instrument's response function must precede using it to test a theory. We report SIVE (Synthetic Instrument Validation Experiment): a fictional municipality (Montelago) with 120 synthetic personas of known latent structure, exposed to seven conditions spanning strongly positive to strongly negative institutional communications about a water network. Seven pre-registered criteria, evaluated across a temperature sweep, jointly assess fidelity, stability, noise floor, specificity, sensitivity, and ordering. All seven pass at every temperature. A central finding turns a calibration failure into a diagnostic success: a message designed as "weakly positive" was identified by the instrument as functionally negative, traced to unresolved problems, uncertainty, and institutional passivity in its text; a redesigned version restored the expected ordering and interacts with agents' latent trust in unanticipated ways. A noise sub-experiment shows the instrument's intrinsic noise is roughly half the cross-agent estimate and stable across temperatures. Individual trajectories reveal coherent micro-dynamics that summary statistics obscure. Full data are available via an interactive explorer.

Problem

• controllability
• synthetic population
• internal validity
• LLM agents
• instrument calibration

Innovation

• controllability
• synthetic population
• LLM agents
• instrument validation
• latent structure