Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

2025-11-19

AI Summary

This paper addresses the coupled multi-agent reinforcement learning (CMARL) problem with individual safety constraints, where agent policies depend on both local states and neighboring states as well as parameters. To tackle the challenges of decentralized cooperative optimization and safety-critical control, we propose a distributed primal-dual algorithm: it employs a κₚ-hop coupled policy and an independent time-varying communication network to avoid direct sharing of policy parameters and Lagrange multipliers; integrates local variable estimation, truncated neighborhood rewards, incomplete information exchange, and multi-step neighbor aggregation. Theoretically, the algorithm converges with high probability to an ε-first-order stationary point, with approximation error bounded by 𝒪(γ^((κ+1)/κₚ)). Experiments on GridWorld demonstrate its effectiveness in achieving safe, scalable, and cooperative optimization under safety constraints.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In this work, we investigate constrained multi-agent reinforcement learning (CMARL), where agents collaboratively maximize the sum of their local objectives while satisfying individual safety constraints. We propose a framework where agents adopt coupled policies that depend on both local states and parameters, as well as those of their $kappa_p$-hop neighbors, with $kappa_p>0$ denoting the coupling distance. A distributed primal-dual algorithm is further developed under this framework, wherein each agent has access only to state-action pairs within its $2kappa_p$-hop neighborhood and to reward information within its $kappa + 2kappa_p$-hop neighborhood, with $kappa>0$ representing the truncation distance. Moreover, agents are not permitted to directly share their true policy parameters or Lagrange multipliers. Instead, each agent constructs and maintains local estimates of these variables for other agents and employs such estimates to execute its policy. Additionally, these estimates are further updated and exchanged exclusively through an independent, time-varying networks, which enhances the overall system security. We establish that, with high probability, our algorithm can achieve an $epsilon$-first-order stationary convergence with an approximation error of $mathcal{O}(gamma^{frac{kappa+1}{kappa_{p}}})$ for discount factor $gammain(0,1)$. Finally, simulations in GridWorld environment are conducted to demonstrate the effectiveness of the proposed algorithm.

Problem

• Maximizing local objectives while satisfying individual safety constraints in multi-agent systems
• Developing distributed algorithms with limited neighbor information access under coupled policies
• Achieving secure coordination without direct sharing of policy parameters or multipliers

Innovation

• Distributed primal-dual algorithm for constrained multi-agent reinforcement learning
• Agents use coupled policies with local and neighbor information
• Local estimates of parameters exchanged via time-varying networks