Multi-Agent Reinforcement Learning via Agent-Specific Preference

2026-08-09

AI Summary

This work addresses the challenge of designing a global reward function in heterogeneous multi-agent reinforcement learning that effectively accommodates diverse agent objectives. To this end, the authors propose MAGPIE, a method that leverages expert preference signals to construct individual reward models for each agent without requiring a predefined global reward. A monotonic aggregation mechanism is introduced to combine these agent-specific rewards into a unified global objective for policy optimization. Theoretically, this approach uniquely bridges agent-specific preference modeling with Nash equilibrium optimization, proving that decentralized preference learning converges to a Nash equilibrium. Empirical results demonstrate that MAGPIE achieves performance on par with handcrafted reward functions across standard multi-agent benchmarks and sequential production-line scenarios, validating its effectiveness in eliminating the need for precise reward engineering.

Abstract

Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized preferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical.

Problem

• Multi-agent reinforcement learning
• heterogeneous agents
• global reward function
• reward engineering
• preference modeling

Innovation

• Multi-Agent Reinforcement Learning
• Preference-Based Learning
• Nash Equilibrium
• Decentralized Reward Modeling
• Monotonic Aggregation