On the Gradient Domination of the LQG Problem

2025-07-11

AI Summary

Policy gradient (PG) methods for the linear-quadratic-Gaussian (LQG) regulator suffer from a lack of gradient dominance and provable global convergence under classical dynamic controller parameterizations. Method: We propose a novel “history-based parameterization” leveraging historical input-output data, enabling gradient dominance and approximate smoothness of the cost function within the LQG framework. This is achieved by combining historical state lifting with geometric analysis of the stable controller set. Contribution/Results: We establish, for the first time in LQG, rigorous global convergence and per-iteration stability guarantees for PG algorithms—both in model-based and model-free settings. Theoretical analysis and numerical experiments demonstrate robust convergence across varying history lengths, even for open-loop unstable systems, thereby overcoming fundamental limitations of conventional parameterizations.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We consider solutions to the linear quadratic Gaussian (LQG) regulator problem via policy gradient (PG) methods. Although PG methods have demonstrated strong theoretical guarantees in solving the linear quadratic regulator (LQR) problem, despite its nonconvex landscape, their theoretical understanding in the LQG setting remains limited. Notably, the LQG problem lacks gradient dominance in the classical parameterization, i.e., with a dynamic controller, which hinders global convergence guarantees. In this work, we study PG for the LQG problem by adopting an alternative parameterization of the set of stabilizing controllers and employing a lifting argument. We refer to this parameterization as a history representation of the control input as it is parameterized by past input and output data from the previous p time-steps. This representation enables us to establish gradient dominance and approximate smoothness for the LQG cost. We prove global convergence and per-iteration stability guarantees for policy gradient LQG in model-based and model-free settings. Numerical experiments on an open-loop unstable system are provided to support the global convergence guarantees and to illustrate convergence under different history lengths of the history representation.

Problem

• LQG problem lacks gradient dominance in classical parameterization
• Study PG for LQG using alternative controller parameterization
• Establish gradient dominance and smoothness for LQG cost

Innovation

• Alternative parameterization of stabilizing controllers
• History representation using past input-output data
• Gradient dominance and approximate smoothness establishment