On the System Theoretic Offline Learning of Continuous-Time LQR with Exogenous Disturbances

2025-09-20

AI Summary

This paper addresses the offline learning problem for continuous-time linear quadratic regulators (LQR) under uncertain external disturbances, distinguishing between exogenous disturbances that are measurable (enabling deterministic estimation) and those that are unmeasurable (stochastic and unknown). We propose a unified offline learning framework integrating adaptive dynamic programming, Lyapunov stability analysis, and sample-based Markov decision process (MDP) approximation. Without requiring online system interaction, the framework guarantees global asymptotic stability and convergence of the learned control gain. Our key contribution is the first rigorously justified offline learning mechanism for continuous-time LQR under non-observable stochastic disturbances—achieving both theoretical soundness grounded in control theory and algorithmic simplicity. Numerical experiments demonstrate the method’s robustness and learning efficiency across diverse disturbance classes.

Technology Category

• Machine Learning: Online Learning & Bandits
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We analyze offline designs of linear quadratic regulator (LQR) strategies with uncertain disturbances. First, we consider the scenario where the exogenous variable can be estimated in a controlled environment, and subsequently, consider a more practical and challenging scenario where it is unknown in a stochastic setting. Our approach builds on the fundamental learning-based framework of adaptive dynamic programming (ADP), combined with a Lyapunov-based analytical methodology to design the algorithms and derive sample-based approximations motivated from the Markov decision process (MDP)-based approaches. For the scenario involving non-measurable disturbances, we further establish stability and convergence guarantees for the learned control gains under sample-based approximations. The overall methodology emphasizes simplicity while providing rigorous guarantees. Finally, numerical experiments focus on the intricacies and validations for the design of offline continuous-time LQR with exogenous disturbances.

Problem

• Designing offline LQR strategies with uncertain exogenous disturbances
• Handling both measurable and unknown disturbances in stochastic settings
• Providing stability guarantees for learned control gains under approximations

Innovation

• Combines adaptive dynamic programming with Lyapunov analysis
• Develops sample-based approximations from MDP approaches
• Provides stability guarantees for non-measurable disturbances