Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

2026-05-03

AI Summary

This work addresses the challenge of balancing exploration intensity in cooperative multi-agent reinforcement learning, where excessive exploration disrupts coordination while insufficient exploration fails to uncover rare yet highly effective strategies; moreover, agents exhibit significant disparities in the reliability of their intrinsic reward signals. To this end, the paper proposes a two-tier exploration regulation framework: globally, a Return-Conditioned Beta (RCB) scheduler dynamically modulates overall exploration intensity, while locally, a Reward Signal Quality (RSQ) metric—based on signal-to-noise ratio—adaptively allocates exploration budgets to prioritize agents with more reliable signals. The approach leverages successor distance as an intrinsic reward, benefiting from its convergence guarantees and order-preserving properties. Evaluated across seven cooperative benchmarks—including MPE, SMAX, and MABrax—the method achieves state-of-the-art performance, significantly outperforming existing approaches.

Abstract

Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity $β$, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting $β$ globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.

Problem

• multi-agent reinforcement learning
• exploration budget allocation
• intrinsic motivation
• coordination collapse
• reward signal quality

Innovation

• Quality-Aware Exploration
• Reward Signal Quality
• Return-Conditioned Scheduling
• Successor Distance
• Multi-Agent Reinforcement Learning