hjb controller synthesis

Existing Hamilton–Jacobi–Bellman (HJB)-driven reinforcement learning methods require full knowledge of system dynamics, rendering them inapplicable to model-free stochastic continuous-time systems. This paper proposes the first fully model-free, HJB-guided temporal-difference (TD) framework that directly approximates the HJB partial differential equation on stochastic differential equation (SDE) systems without accessing drift/diffusion coefficients. Leveraging Itô’s lemma and stochastic calculus, our method establishes a model-free gradient update rule and provides rigorous convergence guarantees. It unifies stochastic optimal control with model-free RL, overcoming the long-standing requirement in HJB-RL for exact dynamical knowledge. Experiments demonstrate substantial improvements in policy performance and sample efficiency across multiple continuous-control benchmarks, with greater stability and computational efficiency compared to transition-kernel-based approaches.

High-dimensional stochastic optimal control remains challenging due to the curse of dimensionality and limitations of conventional approaches relying on probabilistic representations of the Hamilton–Jacobi–Bellman (HJB) equation. Method: This paper proposes a physics-informed deep learning framework grounded in a pathwise HJB operator, unifying modeling and solution. It introduces the pathwise HJB operator as a novel physical constraint and designs two tailored numerical schemes—accommodating both explicit and implicit optimal control structures—integrated with PINNs, dynamic programming, SDE discretization, and HJB path-integral representations. Contribution/Results: A unified theoretical analysis quantifies truncation, approximation, and optimization errors. The framework significantly improves control accuracy and generalization across diverse high-dimensional tasks while ensuring interpretability and analytical tractability, establishing a new paradigm for real-time optimal control of complex stochastic systems.

This paper addresses the optimal feedback control problem for stochastic control-affine systems with unknown nonlinear dynamics and stage-cost functions, leveraging only known control penalty terms and constraints. We propose a fully data-driven framework that innovatively applies kernel mean embedding (KME) to nonparametrically identify the Markov transition operator of controlled diffusion processes, and integrates convex operator theory to reformulate the Hamilton–Jacobi–Bellman (HJB) equation—thereby circumventing the curse of dimensionality inherent in classical dynamic programming. Our approach operates entirely within reproducing kernel Hilbert spaces, utilizing kernel methods and convex optimization without requiring model assumptions or function approximators. Evaluated on multiple high-dimensional nonlinear stochastic systems, the method demonstrates superior data efficiency and scalability. It establishes a novel paradigm for real-time optimal control of black-box systems.

This paper addresses the infinite-horizon optimal closed-loop control problem for nonlinear systems with unknown dynamics, aiming to minimize a given cost function from arbitrary initial states without relying on an explicit system model. We propose a data-driven policy optimization method that integrates the Koopman operator with an actor-critic framework: the Koopman operator enables model-free dynamical representation and differentiable cost gradient estimation, while a parameterized policy is updated via stochastic gradient descent. To our knowledge, this is the first model-free policy gradient method with theoretically guaranteed convergence. Experiments demonstrate stable convergence across multiple nonlinear systems, with control performance significantly surpassing standard model-free reinforcement learning algorithms and closely approaching the optimal benchmark achievable under full model knowledge.

This work addresses the inverse problem of mean-field games (MFGs): reconstructing the obstacle function from partially observed value functions. To overcome the computational intractability of solving the coupled nonlinear forward–backward PDE system inherent in the forward MFG, we propose— for the first time—a decoupled framework based on policy iteration. Our method alternates between solving a linear PDE and a regularized linear inverse problem, inheriting the semantic structure of fixed-point iteration and provably achieving linear convergence. Numerical experiments in 1D and 2D using finite-difference discretization demonstrate that our approach significantly outperforms direct least-squares methods in accuracy, computational efficiency, robustness to observation noise, and scalability. To the best of our knowledge, this is the first inverse MFG solver that simultaneously offers rigorous theoretical guarantees and practical efficacy.

This work addresses the lack of continuous-time Q-learning theory for entropy-regularized mean-field control problems with common noise. By adopting a relaxed control framework, the authors formulate an exploratory Hamilton–Jacobi–Bellman (HJB) equation and introduce an integrated q-function (Iq-function), proving that the optimal policy can be characterized as a bilevel fixed point of its argmax operator. This approach effectively circumvents the policy iteration complexities induced by common noise. In discrete action spaces, the exploratory value function is shown to converge to the relaxed control value function, and the existence and uniqueness of an optimal one-step policy iteration are established. Notably, in the linear–quadratic (LQ) setting, the optimal policy is explicitly derived in Gaussian form, thereby laying the first theoretical foundation for continuous-time mean-field Q-learning with common noise.

This study addresses the absence of implementable Q-learning algorithms for continuous-time mean-field control problems involving controlled common noise. Building upon a relaxed control framework, the authors establish a dynamic relationship between the value function and the Q-function using martingale orthogonality conditions, and approximate unobservable quantities via observable exploratory data. They innovatively propose an Actor-Critic Q-learning algorithm: in the Critic step, function estimates are updated leveraging the martingale condition, while the Actor step iteratively optimizes the policy. This work presents the first Q-learning approach based solely on observable data for this class of problems and proves the convergence of the Actor iteration in the linear-quadratic (LQ) infinite-horizon setting. Numerical experiments on both LQ and non-LQ examples demonstrate the algorithm’s effectiveness and broad applicability.

This work addresses the computational challenges in non-Markovian stochastic optimal control, where the value function is governed by a stochastic Hamilton–Jacobi–Bellman (SHJB) equation whose measurability-induced randomness impedes tractable solution. Under the assumption of control-independent stochastic integral coefficients, the paper introduces the first policy iteration framework tailored to semilinear SHJB equations. The approach iteratively linearizes the original problem into a sequence of linear equations, which are then solved using tools from stochastic analysis and functional approximation. Theoretical analysis establishes that the resulting sequence of approximations converges monotonically in the mean-square sense and exhibits exponential convergence rates, thereby significantly enhancing both the feasibility and efficiency of computing value functions in non-Markovian settings.

This work addresses the challenge of characterizing optimal firm behavior in nonlinear stochastic market environments, where traditional Hamilton–Jacobi–Bellman (HJB) methods are often intractable. The authors introduce a Euclidean path integral control framework that reformulates the equilibrium problem as a forward-looking Lagrangian stochastic control system. By leveraging Itô processes and integrating factors, the approach directly generates optimal strategies without explicitly constructing a value function. For the first time within this framework, a non-cooperative feedback Nash equilibrium is derived and contrasted with mean-field game solutions, revealing fundamental differences from the Pontryagin maximum principle. Combining the Feynman–Kac representation with mean-field approximations, the method yields computationally tractable equilibria for large-scale stochastic markets, with numerical examples demonstrating both its efficacy and its marked divergence from classical HJB solutions.

This paper addresses the time-inconsistent mean-variance optimal stopping problem, where standard dynamic programming fails to characterize equilibrium strategies. To overcome this, we propose a vanishing entropy regularization: a small entropy parameter is introduced to formulate a regularized game, yielding an extended Hamilton–Jacobi–Bellman (HJB) system featuring quadratic terms; taking the vanishing-parameter limit yields a parabolic variational inequality system that rigorously characterizes the equilibrium stopping intensity of the original problem. Theoretically, we establish existence of classical solutions on short time intervals and prove a verification theorem ensuring one-to-one correspondence between solutions and equilibrium strategies. Methodologically, we integrate Cox process modeling, contraction mapping arguments, and variational analysis to construct a rigorous convergence path from the regularized to the original problem. This framework provides the first mathematically tractable characterization of time inconsistency in mean-variance stopping problems.