Resolving Conflicting Constraints in Multi-Agent Reinforcement Learning with Layered Safety

2025-05-04

AI Summary

Multi-agent reinforcement learning (MARL) policies for multi-robot navigation often lack formal safety guarantees and struggle to resolve constraint conflicts under dynamic, high-density conditions. Method: This paper proposes a hierarchical safety framework comprising three tightly integrated components: (i) a low-interaction navigation policy trained via MAPPO; (ii) real-time conflict identification and prioritization using Hamilton–Jacobi reachability analysis and nonlinear control barrier functions (CBFs); and (iii) a safety filter enforcing online tactical corrections. Contribution/Results: The framework unifies policy learning, conflict urgency assessment, and provably correct safety enforcement within a single reachability-based paradigm—overcoming limitations of monolithic CBF approaches or unsafe MARL in strongly coupled multi-agent settings. Evaluated on Crazyflie quadcopter swarms and high-density autonomous aerial mobility (AAM) simulations, it achieves a 99.3% collision avoidance rate, reduces conflicts by 76%, and incurs only <4% overhead in path length and mission time.

Technology Category

• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Multiagent Systems: Multiagent Planning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Preventing collisions in multi-robot navigation is crucial for deployment. This requirement hinders the use of learning-based approaches, such as multi-agent reinforcement learning (MARL), on their own due to their lack of safety guarantees. Traditional control methods, such as reachability and control barrier functions, can provide rigorous safety guarantees when interactions are limited only to a small number of robots. However, conflicts between the constraints faced by different agents pose a challenge to safe multi-agent coordination. To overcome this challenge, we propose a method that integrates multiple layers of safety by combining MARL with safety filters. First, MARL is used to learn strategies that minimize multiple agent interactions, where multiple indicates more than two. Particularly, we focus on interactions likely to result in conflicting constraints within the engagement distance. Next, for agents that enter the engagement distance, we prioritize pairs requiring the most urgent corrective actions. Finally, a dedicated safety filter provides tactical corrective actions to resolve these conflicts. Crucially, the design decisions for all layers of this framework are grounded in reachability analysis and a control barrier-value function-based filtering mechanism. We validate our Layered Safe MARL framework in 1) hardware experiments using Crazyflie drones and 2) high-density advanced aerial mobility (AAM) operation scenarios, where agents navigate to designated waypoints while avoiding collisions. The results show that our method significantly reduces conflict while maintaining safety without sacrificing much efficiency (i.e., shorter travel time and distance) compared to baselines that do not incorporate layered safety. The project website is available at href{https://dinamo-mit.github.io/Layered-Safe-MARL/}{[this https URL]}

Problem

• Ensuring collision-free navigation in multi-robot systems using MARL
• Resolving conflicting safety constraints among multiple interacting agents
• Integrating layered safety filters with MARL for guaranteed safe coordination

Innovation

• Combines MARL with layered safety filters
• Prioritizes urgent corrective actions
• Uses reachability analysis for safety