TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning

2026-06-16

AI Summary

This work addresses the challenge of safe coordination in multi-agent reinforcement learning, where hybrid discrete-continuous action spaces, hard safety constraints during training, and physical dynamics are tightly coupled, leading to persistent bias cycles. To break this triple coupling, we propose TRIDENT, a novel framework that jointly designs three core components: Richardson–Romberg gradient correction, Lyapunov-constrained trust region updates, physics-informed residual value decomposition, and Gumbel–Softmax bias suppression. Evaluated on tasks such as multi-UAV edge computing, TRIDENT reduces training-time constraint violations by 95.5% and improves rewards by 13.5%. Theoretically, it guarantees a convergence rate of $\widetilde{O}(1/\sqrt{K})$ and a cumulative violation bound of $O(\sqrt{K})$, thereby achieving provably safe multi-agent learning.

Abstract

Safe coordination in networked cyber-physical systems forces learning algorithms to simultaneously handle hybrid discrete-continuous actions, hard training-time safety constraints, and physics-governed dynamics. We show that these three features form a directed cycle of biases that defeats any naive composition of off-the-shelf modules, and formalize this as a three-way coupling lemma. We then introduce TRIDENT, the first MARL framework whose three components are co-designed to cancel each leak: a Richardson-Romberg gradient correction reducing Gumbel-Softmax bias from O(tau) to O(tau^2), a Lyapunov-constrained sequential trust-region update enforcing per-iterate feasibility, and a physics-informed residual critic that decomposes value rather than reward. We prove an O~(1/sqrt(K)) convergence rate to a constrained Nash equilibrium and an O(sqrt(K)) cumulative-violation bound. On multi-UAV mobile-edge computing, autonomous intersection management, and a hybrid SMAC variant, TRIDENT cuts training-time violations by 95.5% over MADDPG and 76.3% over MACPO, while improving reward by 13.5% over the strongest unconstrained baseline.

Problem

• hybrid actions
• safety constraints
• physics-governed dynamics
• multi-agent reinforcement learning
• safe coordination

Innovation

• multi-agent reinforcement learning
• safety constraints
• hybrid action spaces
• physics-informed learning
• constrained Nash equilibrium