Learning Coordinated Preference for Multi-Objective Multi-Agent Reinforcement Learning

2026-06-12

AI Summary

This work addresses the coordination challenges in cooperative multi-objective multi-agent reinforcement learning arising from conflicting objectives and agent heterogeneity. To tackle this, we propose Preference-Coordinated Multi-Agent Policy Optimization (PCMA), which learns agent-specific coordination preferences formulated within a team-optimal game framework to achieve complementary trade-offs. Our approach innovatively incorporates a preference diversity mechanism and provides theoretical guarantees that first-order improvement decomposition enhances overall team performance. Experimental results demonstrate that PCMA significantly outperforms baseline methods across multiple cooperative multi-objective environments and real-world traffic control scenarios, effectively improving both system performance and coordination capability.

Abstract

Cooperative multi-objective multi-agent reinforcement learning (MOMARL) models team decision making under multiple, potentially conflicting objectives. In this setting, conflicts arise not only across objectives but also across agents with different observations, roles, and contributions. We propose Preference Coordinated Multi-agent Policy Optimization (PCMA), which learns coordinated agent-specific preferences to enable complementary trade-offs among agents. Theoretically, we formulate cooperative MOMARL as a team-optimal game and show that, under suitable conditions, preference diversity can induce team improvement through a first-order improvement decomposition. Experiments on multiple cooperative MOMA environments and a practical traffic-control scenario show that PCMA improves both performance and trade-off coordination.

Problem

• multi-objective
• multi-agent
• preference coordination
• cooperative reinforcement learning
• conflicting objectives

Innovation

• Preference Coordination
• Multi-Objective Multi-Agent Reinforcement Learning
• Team-Optimal Game
• Complementary Trade-offs
• Policy Optimization