Concept Learning for Cooperative Multi-Agent Reinforcement Learning

2025-07-27

AI Summary

In multi-agent reinforcement learning (MARL), the black-box nature of neural networks impedes transparency of cooperative mechanisms, undermining model interpretability and interoperability. To address this, we propose Conceptual Multi-agent Q-learning (CMQ), the first method to explicitly model human-understandable cooperation concepts as supervised concept vectors, thereby establishing a concept-bottleneck-based value decomposition framework that jointly achieves interpretability and high performance. CMQ integrates global state embedding, supervised concept learning, and a value decomposition architecture. It substantially outperforms state-of-the-art methods on StarCraft II and the Level-Based Foraging (LBF) benchmarks. Crucially, it supports test-time concept intervention, enabling detection of cooperative biases and spurious artifacts, as well as dynamic concept analysis and controllable policy modulation. Our core contribution is the pioneering integration of cooperative semantics into the MARL value decomposition paradigm—uniquely reconciling interpretability, controllability, and strong empirical performance.

Technology Category

• Multiagent Systems: Multiagent Learning
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Quantum Machine Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite substantial progress in applying neural networks (NN) to multi-agent reinforcement learning (MARL) areas, they still largely suffer from a lack of transparency and interoperability. However, its implicit cooperative mechanism is not yet fully understood due to black-box networks. In this work, we study an interpretable value decomposition framework via concept bottleneck models, which promote trustworthiness by conditioning credit assignment on an intermediate level of human-like cooperation concepts. To address this problem, we propose a novel value-based method, named Concepts learning for Multi-agent Q-learning (CMQ), that goes beyond the current performance-vs-interpretability trade-off by learning interpretable cooperation concepts. CMQ represents each cooperation concept as a supervised vector, as opposed to existing models where the information flowing through their end-to-end mechanism is concept-agnostic. Intuitively, using individual action value conditioning on global state embeddings to represent each concept allows for extra cooperation representation capacity. Empirical evaluations on the StarCraft II micromanagement challenge and level-based foraging (LBF) show that CMQ achieves superior performance compared with the state-of-the-art counterparts. The results also demonstrate that CMQ provides more cooperation concept representation capturing meaningful cooperation modes, and supports test-time concept interventions for detecting potential biases of cooperation mode and identifying spurious artifacts that impact cooperation.

Problem

• Interpretable cooperation concepts in MARL
• Overcoming performance-vs-interpretability trade-off
• Detecting biases in cooperation modes

Innovation

• Interpretable value decomposition via concept bottleneck
• Learning cooperation concepts with supervised vectors
• Action value conditioning on global state embeddings