Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia
Existing evaluation methods inadequately measure large language model (LLM) agents’ cooperative generalization capability in novel, mixed-motive social scenarios. Method: We conduct a systematic zero-shot evaluation on the Concordia multi-agent simulation platform, assessing LLM agents’ ability to recognize and realize mutual benefit across diverse social interaction tasks—including negotiation and collective action—using a novel quantitative framework for general cooperative intelligence. This framework emphasizes high-generalization dimensions such as persuasion and norm enforcement. Contribution/Results: Empirical analysis of NeurIPS 2024 Concordia Competition data reveals substantial limitations in current LLM agents’ cross-context cooperative generalization, particularly in dynamic coordination and implicit norm modeling. Our work establishes a new paradigm for benchmarking and diagnosing cooperative intelligence, advancing both methodological rigor and diagnostic precision in multi-agent cooperation research.