Path Integral Value Matching for Linear Quadratic Stochastic Optimal Control

2026-08-11

AI Summary

This work addresses the high computational cost, reliance on full-trajectory rollouts, and training instability inherent in existing policy-based linear-quadratic stochastic optimal control methods by introducing a novel value-function-based paradigm. By truncating and marginalizing path integral control, the authors derive, for the first time, a temporally recursive formulation of the value function. This formulation enables efficient off-policy training through temporal difference learning combined with Girsanov’s theorem. The proposed approach overcomes the high-variance bottleneck of conventional path integral control, substantially improving training stability and sample efficiency. Empirical results demonstrate that the method achieves state-of-the-art accuracy in low-dimensional tasks while offering an order-of-magnitude gain in computational efficiency; in high-dimensional settings, it effectively mitigates mode collapse and significantly enhances scalability.

Abstract

Linear Quadratic Stochastic Optimal Control (LQ-SOC) establishes a fundamental framework for steering noisy dynamical systems and has recently gained renewed interest in the machine learning community. However, current state-of-the-art policy-based methods suffer from prohibitive computational costs and instability due to their heavy reliance on full-trajectory simulation. To overcome these limitations, we propose a paradigm shift toward a value-based approach by revisiting Path Integral Control (PIC). Although standard PIC suffers from the same high-variance bottleneck as policy-based methods, we discover that by truncating and marginalizing the original path integral formulation, we can derive a temporal recursive form of the value function. Building upon this theoretical foundation, we propose the Path Integral Value Matching (PI-VM) algorithm. Specifically, we employ temporal-difference learning to approximate the recursive value dynamics, and further integrate the Girsanov theorem with experience replay to enable off-policy training. We benchmark PI-VM against SOTA policy-based methods across various SOC benchmarks and sampling tasks. Empirical results demonstrate that PI-VM matches SOTA precision with an order-of-magnitude efficiency gain in low-dimensional settings, while effectively mitigating mode collapse in high-dimensional scenarios. Consequently, PI-VM offers a scalable solution for solving complex SOC problems.

Problem

• Linear Quadratic Stochastic Optimal Control
• Path Integral Control
• policy-based methods
• computational cost
• instability

Innovation

• Path Integral Control
• Value-based Reinforcement Learning
• Temporal Difference Learning
• Girsanov Theorem
• Off-policy Training