Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning

2025-09-11

AI Summary

Existing continuous-time multi-agent reinforcement learning (CTRL) methods struggle with scalability in complex dynamical systems exhibiting high-frequency or irregular temporal interactions, primarily due to the curse of dimensionality and challenges in centralized value function approximation under discrete-time modeling constraints. This paper introduces the first scalable CTRL framework for multi-agent settings. First, it proposes a value-gradient iteration module that jointly optimizes the value function and its spatiotemporal gradients, enhancing accuracy in solving the Hamilton–Jacobi–Bellman (HJB) equation. Second, it employs physics-informed neural networks (PINNs) to approximate the viscosity solution of the HJB equation, integrating continuous-time value iteration with a value-gradient alignment mechanism for iterative refinement along trajectories. Third, extensive experiments on multi-agent particle environments and MuJoCo benchmarks demonstrate significant performance gains over prior CTRL approaches, validating both high-dimensional scalability and policy stability.

Technology Category

• Multiagent Systems: Multiagent Learning
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing reinforcement learning (RL) methods struggle with complex dynamical systems that demand interactions at high frequencies or irregular time intervals. Continuous-time RL (CTRL) has emerged as a promising alternative by replacing discrete-time Bellman recursion with differential value functions defined as viscosity solutions of the Hamilton--Jacobi--Bellman (HJB) equation. While CTRL has shown promise, its applications have been largely limited to the single-agent domain. This limitation stems from two key challenges: (i) conventional solution methods for HJB equations suffer from the curse of dimensionality (CoD), making them intractable in high-dimensional systems; and (ii) even with HJB-based learning approaches, accurately approximating centralized value functions in multi-agent settings remains difficult, which in turn destabilizes policy training. In this paper, we propose a CT-MARL framework that uses physics-informed neural networks (PINNs) to approximate HJB-based value functions at scale. To ensure the value is consistent with its differential structure, we align value learning with value-gradient learning by introducing a Value Gradient Iteration (VGI) module that iteratively refines value gradients along trajectories. This improves gradient fidelity, in turn yielding more accurate values and stronger policy learning. We evaluate our method using continuous-time variants of standard benchmarks, including multi-agent particle environment (MPE) and multi-agent MuJoCo. Our results demonstrate that our approach consistently outperforms existing continuous-time RL baselines and scales to complex multi-agent dynamics.

Problem

• Extends continuous-time RL to multi-agent systems
• Overcomes curse of dimensionality in HJB equations
• Improves value function approximation for stability

Innovation

• Physics-informed neural networks approximate HJB value functions
• Value Gradient Iteration refines gradients along trajectories
• Continuous-time framework scales to multi-agent dynamics