On the Fundamental Limitations of Decentralized Learnable Reward Shaping in Cooperative Multi-Agent Reinforcement Learning

2025-10-27

AI Summary

This work investigates fundamental limitations of decentralized learnable reward shaping in cooperative multi-agent reinforcement learning (MARL). We propose DMARL-RSA, a framework wherein agents independently learn reward-shaping functions, and conduct systematic evaluation in the simple_spread_v3 environment. Our analysis identifies three core bottlenecks: non-stationarity in policy updates, inaccurate credit assignment, and misalignment between individual and global objectives—leading to a coordination paradox between local optimization and global performance. Empirical results show DMARL-RSA achieves an average reward of −24.20, substantially underperforming centralized MAPPO (1.92) and approaching the level of independent PPO (IPPO), thereby confirming that current decentralized reward learning cannot overcome intrinsic coordination constraints. The key contribution is the first formal characterization and empirical validation of structural limitations inherent to decentralized learnable reward shaping in MARL.

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Coordination and Collaboration
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Recent advances in learnable reward shaping have shown promise in single-agent reinforcement learning by automatically discovering effective feedback signals. However, the effectiveness of decentralized learnable reward shaping in cooperative multi-agent settings remains poorly understood. We propose DMARL-RSA, a fully decentralized system where each agent learns individual reward shaping, and evaluate it on cooperative navigation tasks in the simple_spread_v3 environment. Despite sophisticated reward learning, DMARL-RSA achieves only -24.20 +/- 0.09 average reward, compared to MAPPO with centralized training at 1.92 +/- 0.87--a 26.12-point gap. DMARL-RSA performs similarly to simple independent learning (IPPO: -23.19 +/- 0.96), indicating that advanced reward shaping cannot overcome fundamental decentralized coordination limitations. Interestingly, decentralized methods achieve higher landmark coverage (0.888 +/- 0.029 for DMARL-RSA, 0.960 +/- 0.045 for IPPO out of 3 total) but worse overall performance than centralized MAPPO (0.273 +/- 0.008 landmark coverage)--revealing a coordination paradox between local optimization and global performance. Analysis identifies three critical barriers: (1) non-stationarity from concurrent policy updates, (2) exponential credit assignment complexity, and (3) misalignment between individual reward optimization and global objectives. These results establish empirical limits for decentralized reward learning and underscore the necessity of centralized coordination for effective multi-agent cooperation.

Problem

• Evaluating decentralized learnable reward shaping effectiveness in cooperative multi-agent reinforcement learning
• Investigating coordination limitations between local optimization and global performance objectives
• Identifying fundamental barriers to decentralized reward learning in multi-agent cooperation

Innovation

• Decentralized learnable reward shaping for agents
• Individual reward learning in cooperative navigation
• Identifies barriers in multi-agent coordination