Exploratory Randomization for Discrete-Time Linear Exponential Quadratic Gaussian (LEQG) Problem

2025-01-10

AI Summary

This paper addresses exploratory randomization modeling for the discrete-time linear exponential quadratic Gaussian (LEQG) control problem, motivated by risk-sensitive portfolio management. Methodologically, it introduces stochastic perturbations to the control policy and leverages the duality between free energy and relative entropy to rigorously reformulate the discrete LEQG problem as an entropy-regularized, risk-neutral linear quadratic Gaussian (LQG) problem. Using dynamic programming and stochastic optimal control theory, the authors derive the analytical solution structure of the entropy-regularized formulation and establish an interpretable mapping between exploratory randomization and risk-sensitive control. Key contributions include: (i) a systematic extension of the energy–entropy duality to discrete-time stochastic control, providing a rigorous theoretical foundation for entropy regularization; and (ii) a resulting control policy that simultaneously ensures robustness and generalizability, significantly enhancing decision reliability under uncertainty.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We investigate exploratory randomization for an extended linear-exponential-quadratic-Gaussian (LEQG) control problem in discrete time. This extended control problem is related to the structure of risk-sensitive investment management applications. We introduce exploration through a randomization of the control. Next, we apply the duality between free energy and relative entropy to reduce the LEQG problem to an equivalent risk-neutral LQG control problem with an entropy regularization term, see, e.g. Dai Pra et al. (1996), for which we present a solution approach based on Dynamic Programming. Our approach, based on the energy-entropy duality may also be considered as leading to a justification for the use, in the literature, of an entropy regularization when applying a randomized control.

Problem

• Investigating exploratory randomization in discrete-time LEQG control
• Applying energy-entropy duality to transform risk-sensitive problems
• Providing justification for entropy regularization in randomized control

Innovation

• Randomized control for exploration in LEQG
• Duality transforms problem to risk-neutral LQG
• Dynamic Programming with entropy regularization solution