One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

2026-08-12

AI Summary

This work addresses the issue of policy overfitting and poor generalization in multi-agent reinforcement learning caused by reliance on a single large language model (LLM) as a simulator. The paper formally characterizes, for the first time, the phenomenon of “simulator collapse” and introduces a two-stage solution: Co-Training based on population-level collaboration during training, and Verbalized Sampling during inference to enhance policy generalization to unseen environments and real users. The proposed SCOPE framework achieves substantial performance gains across three multi-turn dialogue benchmarks—Verbalized Sampling alone improves held-out success rates by up to 9%, and combining it with Co-Training further increases this gain to 14%. Human evaluations corroborate these improvements and confirm that policy diversity is effectively preserved.

Abstract

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $τ^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Problem

• simulator collapse
• multi-agent reinforcement learning
• generalization
• mode collapse
• human-AI interaction

Innovation

• simulator collapse
• Verbalized Sampling
• Co-Training
• multi-agent reinforcement learning
• population-based training