Bandwidth-constrained Variational Message Encoding for Cooperative Multi-agent Reinforcement Learning

2025-12-11

AI Summary

To address inefficient coordination communication and performance degradation from dimensionality reduction in bandwidth-constrained multi-agent reinforcement learning (MARL), this paper proposes the Bayesian Variational Message Encoding (BVME) framework. BVME directly incorporates information-theoretic constraints into decision-level representations, enabling interpretable and tunable lossy message compression via KL-divergence-regularized Gaussian variational posteriors, while leveraging graph neural networks to model inter-agent dependencies. Evaluated on SMACv1, SMACv2, and MPE benchmarks, BVME achieves comparable or superior policy performance using 67–83% fewer message dimensions; gains are especially pronounced under extremely sparse communication topologies, with negligible computational overhead. To the best of our knowledge, BVME is the first variational encoding framework in MARL that explicitly unifies rate-distortion optimization with end-to-end policy learning.

Technology Category

• Multiagent Systems: Agent Communication
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Graph-based multi-agent reinforcement learning (MARL) enables coordinated behavior under partial observability by modeling agents as nodes and communication links as edges. While recent methods excel at learning sparse coordination graphs-determining who communicates with whom-they do not address what information should be transmitted under hard bandwidth constraints. We study this bandwidth-limited regime and show that naive dimensionality reduction consistently degrades coordination performance. Hard bandwidth constraints force selective encoding, but deterministic projections lack mechanisms to control how compression occurs. We introduce Bandwidth-constrained Variational Message Encoding (BVME), a lightweight module that treats messages as samples from learned Gaussian posteriors regularized via KL divergence to an uninformative prior. BVME's variational framework provides principled, tunable control over compression strength through interpretable hyperparameters, directly constraining the representations used for decision-making. Across SMACv1, SMACv2, and MPE benchmarks, BVME achieves comparable or superior performance while using 67--83% fewer message dimensions, with gains most pronounced on sparse graphs where message quality critically impacts coordination. Ablations reveal U-shaped sensitivity to bandwidth, with BVME excelling at extreme ratios while adding minimal overhead.

Problem

• Addresses bandwidth-limited message encoding in multi-agent reinforcement learning
• Introduces variational framework for selective compression of communication messages
• Enhances coordination performance under hard bandwidth constraints on sparse graphs

Innovation

• Variational message encoding for bandwidth constraints
• Gaussian posteriors regularized via KL divergence
• Tunable compression control with interpretable hyperparameters