Physics-Informed Multi-Agent Reinforcement Learning for Distributed Multi-Robot Problems

2023-12-30

arXiv.org

AI Summary

To address the scalability limitations of centralized policies and the lack of coordination in independent policies for multi-robot systems, this paper proposes a scalable and physically consistent distributed reinforcement learning framework. Methodologically, it pioneers the integration of port-Hamiltonian system modeling with graph self-attention mechanisms, enabling efficient local information utilization under dynamic interaction graphs while enforcing energy conservation; it employs distributed policy parameterization—bypassing value function decomposition—and explicitly encodes inter-robot dependencies. Key contributions include: (1) a self-attention–enhanced port-Hamiltonian policy representation; (2) zero-shot post-training scalability to robot swarms six times larger than those used during training; and (3) cumulative rewards twice those of prior state-of-the-art methods across multiple tasks, demonstrating significant improvements in scalability and collaborative efficiency.

Technology Category

• Intelligent Robots: Multi-Robot Systems
• Multiagent Systems: Multiagent Learning
• Machine Learning: Distributed Machine Learning & Federated Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The networked nature of multi-robot systems presents challenges in the context of multi-agent reinforcement learning. Centralized control policies do not scale with increasing numbers of robots, whereas independent control policies do not exploit the information provided by other robots, exhibiting poor performance in cooperative-competitive tasks. In this work we propose a physics-informed reinforcement learning approach able to learn distributed multi-robot control policies that are both scalable and make use of all the available information to each robot. Our approach has three key characteristics. First, it imposes a port-Hamiltonian structure on the policy representation, respecting energy conservation properties of physical robot systems and the networked nature of robot team interactions. Second, it uses self-attention to ensure a sparse policy representation able to handle time-varying information at each robot from the interaction graph. Third, we present a soft actor-critic reinforcement learning algorithm parameterized by our self-attention port-Hamiltonian control policy, which accounts for the correlation among robots during training while overcoming the need of value function factorization. Extensive simulations in different multi-robot scenarios demonstrate the success of the proposed approach, surpassing previous multi-robot reinforcement learning solutions in scalability, while achieving similar or superior performance (with averaged cumulative reward up to x2 greater than the state-of-the-art with robot teams x6 larger than the number of robots at training time).

Problem

• Scalable distributed control for multi-robot systems
• Cooperative-competitive task performance improvement
• Physics-informed policy respecting energy conservation

Innovation

• Physics-informed reinforcement learning for distributed control
• Port-Hamiltonian policy structure ensures energy conservation
• Self-attention enables sparse, scalable policy representation