HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

2026-06-27

AI Summary

Existing multi-agent communication methods treat messages as flat vectors decoupled from the structural organization of observations, thereby overlooking the inherent hierarchical structure—such as group–entity relationships—that naturally arises in cooperative settings, which limits both communication efficiency and task performance. To address this, this work proposes HiComm, a receiver-driven, plug-in communication module that explicitly incorporates hierarchical observation structure into the communication mechanism. HiComm enables structured information retrieval through a three-stage differentiable decoding process: group selection → sender selection → entity selection. Discrete choices are optimized end-to-end via Straight-Through Gumbel-Softmax, while a lightweight shared projection design ensures seamless compatibility with mainstream multi-agent reinforcement learning frameworks. Experiments demonstrate that HiComm matches or surpasses state-of-the-art methods across diverse cooperative tasks while reducing per-agent communication overhead by up to 23× per episode.

Abstract

Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose \textsc{HiComm}, a plug-in communication module that grounds messages in the sender's hierarchical observation. \textsc{HiComm} is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that \textsc{HiComm} matches or outperforms representative learned communication baselines while reducing communication volume by up to $23\times$ per receiver per episode.

Problem

• multi-agent reinforcement learning
• communication
• partial observability
• hierarchical observation
• inductive bias

Innovation

• Hierarchical Communication
• Multi-agent Reinforcement Learning
• Structured Information Retrieval
• Receiver-driven Query
• Gumbel-Softmax Selection