R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning

2025-05-30

AI Summary

Existing MARL approaches model roles as static abstractions derived from historical experience, overlooking their causal influence on agents’ future behavioral trajectories—leading to suboptimal coordination efficiency. To address this, we propose a *future-behavior-shaping-driven role discovery* framework, which formally defines roles as causal variables that shape future trajectories and introduces a novel objective: maximizing the triple mutual information among roles, observed trajectories, and future behaviors. Methodologically, our approach integrates contrastive learning for role representation, differentiable dynamics modeling, intrinsic-reward-guided diversity regularization, and optimized mutual information estimation. Evaluated on SMAC and SMACv2 benchmarks, our method achieves up to a 20% improvement in win rate over state-of-the-art baselines, demonstrating substantial gains in generalization and robustness of multi-agent coordination.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multi-agent reinforcement learning (MARL) has achieved significant progress in large-scale traffic control, autonomous vehicles, and robotics. Drawing inspiration from biological systems where roles naturally emerge to enable coordination, role-based MARL methods have been proposed to enhance cooperation learning for complex tasks. However, existing methods exclusively derive roles from an agent's past experience during training, neglecting their influence on its future trajectories. This paper introduces a key insight: an agent's role should shape its future behavior to enable effective coordination. Hence, we propose Role Discovery and Diversity through Dynamics Models (R3DM), a novel role-based MARL framework that learns emergent roles by maximizing the mutual information between agents' roles, observed trajectories, and expected future behaviors. R3DM optimizes the proposed objective through contrastive learning on past trajectories to first derive intermediate roles that shape intrinsic rewards to promote diversity in future behaviors across different roles through a learned dynamics model. Benchmarking on SMAC and SMACv2 environments demonstrates that R3DM outperforms state-of-the-art MARL approaches, improving multi-agent coordination to increase win rates by up to 20%.

Problem

• Enhances multi-agent coordination through dynamic role discovery
• Improves role diversity using mutual information maximization
• Boosts win rates in MARL benchmarks by 20%

Innovation

• Uses dynamics models for role discovery
• Maximizes mutual information for coordination
• Employs contrastive learning for diversity