ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning

2026-05-22

AI Summary

This work addresses the challenge of low learning efficiency in multi-agent reinforcement learning caused by sparse rewards. The authors propose ARMS, a framework that enhances sampling efficiency without altering the underlying game-theoretic structure of agents’ policies. ARMS introduces Nash equilibrium preservation as a novel reward shaping criterion, leveraging conditional best-response analysis to ensure that shaped rewards do not modify agents’ sets of optimal responses. The framework further integrates self-supervised trajectory ranking, cross-agent parameter sharing, and an alternating optimization mechanism between policy and reward modules to distill dense shaping signals from sparse environmental rewards. Experimental results demonstrate that ARMS significantly improves training efficiency in partially observable multi-agent pathfinding tasks, scales effectively to higher degrees of reward sparsity and larger numbers of agents, and exhibits strong generalization across environments.

Abstract

Sparse rewards are a major bottleneck in multi-agent reinforcement learning (MARL), where simultaneous learning induces non-stationarity and makes reward design especially delicate. Reward shaping can accelerate learning, but in the multi-agent setting it must preserve the strategic structure of the problem rather than merely improve short-term optimization. We propose Automatic Reward-shaping in Multi-agent Systems (ARMS), a self-supervised reward shaping framework for MARL that learns dense shaping signals from sparse environmental rewards through trajectory ranking. Since single-agent trajectory-ranking guarantees do not directly transfer to MARL, we reformulate policy invariance through conditional best-response reasoning, and show that if certain conditions hold, then using shaping rewards preserves each agent's best-response set under fixed opponent policies, and consequently preserve the set of Nash equilibria. Guided by this perspective, ARMS alternates between policy learning and reward learning while sharing shaping parameters across agents for efficiency. Experiments in a partially observable multi-agent pathfinding domain show that ARMS improves sampling efficiency under increasing reward sparsity and agent count, generalizes to unseen environments, and reveals a MARL-specific failure mode in which limited exploration and coupled policy--reward dynamics induce oscillatory behavior. Increasing exploration mitigates this effect and stabilizes learning. To the best of our knowledge, ARMS is the first automatic reward shaping framework for MARL whose design is motivated by a game-theoretic equilibrium-preservation result.

Problem

• sparse rewards
• multi-agent reinforcement learning
• reward shaping
• non-stationarity
• Nash equilibria

Innovation

• reward shaping
• multi-agent reinforcement learning
• Nash equilibrium preservation
• trajectory ranking
• self-supervised reward learning