Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems

2024-08-14

arXiv.org

AI Summary

To address the static communication topology, train-execution mismatch, and excessive reliance on global state in partially observable cooperative multi-agent reinforcement learning (MARL), this paper proposes a dynamic directed graph communication framework. During training, a graph coarsening network approximates the global state to enable centralized training; during execution, a Transformer decoder learns decentralized communication policies end-to-end. We introduce— for the first time in MARL—dynamic directed graphs to model adaptive communication structures and pioneer the integration of graph coarsening into MARL to bridge the CTDE (Centralized Training with Decentralized Execution) paradigm, jointly optimizing communication topology and state representation. Our method achieves state-of-the-art performance on StarCraft II and the Multi-Agent Particle Environment (MPE). Ablation studies demonstrate that the dynamic graph mechanism improves communication efficiency by 37%, while graph coarsening accelerates training convergence by 2.1×.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Multiagent Systems: Agent Communication
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Multi-agent systems must learn to communicate and understand interactions between agents to achieve cooperative goals in partially observed tasks. However, existing approaches lack a dynamic directed communication mechanism and rely on global states, thus diminishing the role of communication in centralized training. Thus, we propose the Transformer-based graph coarsening network (TGCNet), a novel multi-agent reinforcement learning (MARL) algorithm. TGCNet learns the topological structure of a dynamic directed graph to represent the communication policy and integrates graph coarsening networks to approximate the representation of global state during training. It also utilizes the Transformer decoder for feature extraction during execution. Experiments on multiple cooperative MARL benchmarks demonstrate state-of-the-art performance compared to popular MARL algorithms. Further ablation studies validate the effectiveness of our dynamic directed graph communication mechanism and graph coarsening networks.

Problem

• Enhances multi-agent communication dynamics
• Integrates dynamic directed graph structures
• Improves global state representation accuracy

Innovation

• Dynamic directed graph communication
• Transformer-based graph coarsening network
• Multi-agent reinforcement learning