Multi-agent cooperation through learning-aware policy gradients

2024-10-24

arXiv.org

AI Summary

Spontaneous cooperation remains fundamentally challenging in self-interested, independently learning multi-agent systems. Method: We propose Learning-Aware Policy Gradient (LAPG), the first unbiased policy gradient method for multi-agent reinforcement learning that avoids higher-order derivatives. LAPG explicitly models opponents’ learning dynamics via history-based inference, integrated with long-context sequence modeling (e.g., Transformers), enabling precise, temporally extended action coordination. Results & Contributions: Evaluated on canonical social dilemmas—including iterated prisoner’s dilemma—LAPG significantly improves cooperation stability and cumulative reward, especially in complex environments requiring long-horizon coordination. Its core contribution is a differentiable learning-dynamics-aware paradigm that mechanistically elucidates how cooperation emerges; it also provides the first higher-order-derivative-free, unbiased multi-agent policy gradient estimator.

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. How can we achieve cooperation among self-interested, independent learning agents? Promising recent work has shown that in certain tasks cooperation can be established between learning-aware agents who model the learning dynamics of each other. Here, we present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning, which takes into account that other agents are themselves learning through trial and error based on multiple noisy trials. We then leverage efficient sequence models to condition behavior on long observation histories that contain traces of the learning dynamics of other agents. Training long-context policies with our algorithm leads to cooperative behavior and high returns on standard social dilemmas, including a challenging environment where temporally-extended action coordination is required. Finally, we derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.

Problem

• Achieving cooperation among self-interested, independent learning agents.
• Developing unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning.
• Explaining how and when cooperation arises among self-interested learning-aware agents.

Innovation

• Unbiased higher-derivative-free policy gradient algorithm
• Efficient sequence models for long observation histories
• Training long-context policies for cooperative behavior