Stabilized neural Hamilton--Jacobi--Bellman solvers: Error analysis and applications in model-based reinforcement learning

2026-05-07

AI Summary

This work addresses the challenges of stability and error control in solving Hamilton-Jacobi-Bellman (HJB) equations for continuous-time reinforcement learning by proposing a mesh-free method that integrates physics-informed neural networks, finite differences interpreted via shift operators, stochastic continuous collocation, and greedy policy improvement. Leveraging a hybrid error analysis framework, the approach explicitly disentangles multiple error sources and quantifies gradient amplification factors, thereby effectively circumventing the implicit viscosity-related blow-up issues prevalent in conventional methods. Experimental results on a 64-dimensional linear-quadratic regulator (LQR) problem and several nonlinear control tasks demonstrate that the proposed method significantly outperforms state-of-the-art model-based and model-free reinforcement learning baselines in terms of residual control, policy alignment, and model error, achieving stable and efficient synthesis of optimal feedback controllers.

Abstract

Physics-informed neural solvers offer a promising route to model-based reinforcement learning in continuous time, where optimal feedback synthesis is governed by Hamilton--Jacobi--Bellman (HJB) equations. Practical implementations often occupy a regime that is neither a classical grid method nor a continuous-PDE PINN: the value function is represented by a neural network, finite-difference HJB policy-evaluation operators are evaluated by network queries at shifted points, and residuals are minimized by random continuous collocation. This regime preserves the stabilized finite-difference policy-evaluation structure while avoiding grid-based value unknowns. We develop an error theory for this hybrid regime. Interpreting finite differences as shift operators acting on neural networks, we prove a population $L^2$ stability estimate for one policy-evaluation step with learned dynamics. The bound separates residual error, initial and exterior-collar mismatch, policy mismatch, and model-identification error, with an explicit gradient amplification factor for learned dynamics, while the underlying linear evaluation stability remains free of hidden inverse-viscosity blow-up. We further give a finite-sample collocation certificate and a conditional multi-step propagation result through greedy policy improvement. Experiments on compact-control LQR upto 64 dimensions, Allen--Cahn control, pendulum, Hopper, and 3D quadrotor benchmarks compare against representative model-based and model-free RL baselines, demonstrating the predicted residual, policy-mismatch, and learned-model error trends.

Problem

• Hamilton--Jacobi--Bellman
• model-based reinforcement learning
• error analysis
• physics-informed neural networks
• policy evaluation

Innovation

• neural HJB solvers
• error analysis
• physics-informed neural networks
• model-based reinforcement learning
• finite-difference shift operators