CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

2026-04-16

AI Summary

Large language models (LLMs) generally exhibit non-cooperative tendencies in social dilemmas, hindering safe and effective multi-agent interactions. This work presents the first systematic comparison—under equilibrium conditions—of four game-theoretic mechanisms for fostering cooperation among LLM agents: repeated interaction, reputation systems, third-party mediation, and contractual agreements. The study finds that contracts and third-party mediation are most effective among strong models, while repeated interaction suffers a significant decline in cooperation rates when opponents change dynamically. Notably, all mechanisms demonstrate enhanced performance under evolutionary pressure favoring individual payoff maximization. These findings provide an empirical foundation and actionable design principles for developing scalable and robust multi-agent cooperative frameworks.

Technology Category

• Multiagent Systems: Mechanism Design
• Game Theory and Economic Paradigms: Cooperative Game Theory
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings. Indeed, our experiments show that recent models -- with or without reasoning enabled -- consistently defect in single-shot social dilemmas. To tackle this safety concern, we present the first comparative study of game-theoretic mechanisms that are designed to enable cooperative outcomes between rational agents _in equilibrium_. Across four social dilemmas testing distinct components of robust cooperation, we evaluate the following mechanisms: (1) repeating the game for many rounds, (2) reputation systems, (3) third-party mediators to delegate decision making to, and (4) contract agreements for outcome-conditional payments between players. Among our findings, we establish that contracting and mediation are most effective in achieving cooperative outcomes between capable LLM models, and that repetition-induced cooperation deteriorates drastically when co-players vary. Moreover, we demonstrate that these cooperation mechanisms become _more effective_ under evolutionary pressures to maximize individual payoffs.

Problem

• LLM agents
• social dilemmas
• cooperation
• safety
• mixed-motive games

Innovation

• CoopEval
• social dilemmas
• cooperation mechanisms
• LLM agents
• game-theoretic equilibrium