model predictive control

This study addresses the lack of a systematic synthesis in research on integrating reinforcement learning (RL) with model predictive control (MPC) for linear systems by proposing the first multidimensional taxonomy tailored to this domain. Drawing on a comprehensive literature review up to 2025, the work establishes a classification framework along five dimensions: RL role, algorithm type, MPC formulation, cost function structure, and application area, followed by an integrative cross-dimensional analysis. The study elucidates representative integration strategies, traces methodological evolution, and identifies key challenges—including computational burden, sample efficiency, robustness, and closed-loop guarantees—thereby offering a structured reference and practical guidance for both theoretical analysis and architectural design in RL–MPC systems.

This work addresses the challenge of real-time robotic arm control in dynamically cluttered environments, where agents must balance rapid responsiveness with foresightful obstacle avoidance to prevent myopic constraint violations. The authors propose a task-space receding horizon controller that generates collision-free terminal pose references through short-horizon, contact-consistent forward simulations respecting non-penetration constraints, then computes only the first-step minimum-acceleration control input that smoothly transitions toward this reference. By integrating the strengths of receding horizon and reactive control, the method efficiently embeds information about contacts, moving obstacles, and self-collisions using inflated convex geometry and an iterative dynamics solver—without requiring full trajectory optimization. Simulations with 40 degrees of freedom demonstrate that a moderate horizon length effectively balances foresight, responsiveness, and computational cost, while hardware experiments on a 6-DOF manipulator confirm strong sim-to-real transfer, outperforming MPC and dynamic optimization fabric approaches in success rate under dynamic clutter while meeting real-time requirements.

This work addresses the limitations of reinforcement learning, which often deviates from human-like behavior, and imitation learning, which struggles to surpass demonstrator performance. The authors propose a behavior-constrained reinforcement learning framework that models short-horizon future trajectories via receding-horizon prediction and conditions policy learning on reference trajectories. Expert behavioral consistency is enforced at the trajectory distribution level rather than through point-wise matching. By integrating receding-horizon credit assignment with behavior constraints, the approach enables robust generalization under disturbances and varying operating conditions. Evaluated in a high-fidelity racing simulator using professional driver data, the trained policies achieve competitive lap times while closely replicating expert driving styles. Human-in-the-loop assessments further confirm the accurate reproduction of tuning-sensitive driving characteristics.

This paper investigates the suboptimality of nominal model-predictive linear-quadratic (LQ) control for unknown linear systems, characterizing a fundamental trade-off among model mismatch, terminal cost approximation error, and prediction horizon length. We develop a novel perturbation analysis framework for the Riccati difference equation, establishing—for the first time—a quantitative relationship between horizon length and the system’s controllability index. Theoretically, we prove that a finite horizon bounded by the controllability index suffices to approximate infinite-horizon optimal performance, and that horizons of length one or infinity are often optimal. Based on this insight, we derive the first adaptive horizon-selection criterion tailored for learning-based control, yielding a tight suboptimality upper bound, an $O(log T)$ regret guarantee, and optimal sample complexity.

This paper addresses online control of dynamic systems under adversarial environments—where cost functions and model disturbances are arbitrarily chosen by an adversary—seeking low regret relative to a benchmark policy class, rather than relying on post-hoc optimality or stochastic noise assumptions as in classical optimal or robust control. We propose the new paradigm of *online nonstochastic control*, which systematically integrates online convex optimization and convex relaxation into the classical control framework for the first time. Our approach abandons probabilistic modeling and instead establishes deterministic, regret-based performance guarantees, coupled with iterative optimization algorithms. We derive finite-time regret bounds and computational complexity analyses. Theoretically, our algorithm achieves provably low regret—$O(sqrt{T})$—while remaining efficiently implementable within standard control settings.

This work addresses the problem of learning non-stationary optimal policies for finite-horizon constrained Markov decision processes (CMDPs), filling a theoretical gap left by prior studies focused on infinite-horizon settings and stationary policies. We propose the first policy gradient algorithm specifically designed for finite-horizon CMDPs, employing time-varying parametric policy networks and Lagrangian relaxation to handle hard constraints. We provide rigorous convergence guarantees—proving that the algorithm converges to a constraint-optimal solution under standard regularity conditions. The method accommodates continuous state-action spaces, supports function approximation, and scales effectively to high-dimensional problems. Empirical evaluation across multiple benchmark tasks demonstrates substantial improvements in cumulative reward, constraint satisfaction rate, and convergence stability, thereby validating both the theoretical assurances and practical efficacy of the approach.

This work investigates how to achieve optimal policies in Markov decision processes (MDPs) that incorporate future information—such as reference trajectories or predictions—by leveraging model predictive control (MPC). The authors formulate MPC as a class of parameterized policies and train them end-to-end via reinforcement learning. Their key contribution lies in establishing, for the first time, the precise structural conditions under which MPC can exactly represent the optimal value function and policy, thereby providing a theoretical foundation for MPC as a structured function approximator with formal guarantees. Empirical validation on a point-mass racing task with future reference trajectories demonstrates that the proposed approach learns policies approaching optimality, confirming its effectiveness.

This work addresses the problem of feedback motion planning for continuous-time stochastic nonlinear systems under Signal Temporal Logic (STL) specifications by proposing a novel framework that integrates predicate erosion with probabilistic reachable tubes. Predicate erosion is employed to transform stochastic STL constraints into tightened deterministic ones, while probabilistic reachable tubes quantify the deviation of stochastic trajectories from their nominal counterparts. Leveraging contraction theory, a tracking controller is designed to establish a closed-loop planning pipeline. The proposed approach significantly reduces the conservatism inherent in conventional methods, achieving high STL satisfaction probability without compromising planning performance. Simulations and real-world experiments on a quadrupedal robot demonstrate that the method outperforms baseline approaches in both STL satisfaction rate and computational efficiency.

This work addresses the challenge of effectively translating social preferences into resource allocation objectives within multi-agent control systems to fulfill ethical and socially responsible missions. By aggregating individual preferences into a welfare-oriented control objective, the study unifies this approach across three major control paradigms: online feedback optimization, Markov decision process control, and model predictive control. It presents the first systematic framework that embeds social welfare principles directly into the control design pipeline, integrating preference aggregation with formal verification mechanisms to yield a certifiably compliant control architecture. This framework offers a novel pathway for automated resource allocation systems that simultaneously ensures fairness, efficiency, and interpretability.

This work addresses the challenge of navigating complex cost landscapes in non-convex model predictive control, where nonlinear dynamics and multiple obstacles often trap gradient-based methods in suboptimal local minima. To overcome this limitation, we propose a Maximum Entropy Differential Dynamic Programming (ME-DDP) framework that integrates deterministic optimization with entropy-maximizing sampling. Our approach employs a two-stage mechanism: it first performs local gradient-based refinement via DDP and then leverages the inverse Hessian of the action-value function to guide policy sampling, enabling escape from local minima and balancing global exploration with local exploitation. We develop three ME-DDP variants, elucidate their theoretical connections to Model Predictive Path Integral (MPPI) control, and demonstrate superior performance across four navigation benchmarks—achieving higher success rates in high-dimensional systems, outperforming MPPI in low-dimensional settings, and exhibiting robustness in real-world quadrotor experiments through dense obstacle fields.

This work addresses the stochastic optimal control problem with joint chance constraints over an infinite horizon. By augmenting the state space, the problem is reformulated as a constrained Markov decision process with an additive structure. The paper establishes strong duality for this setting for the first time, thereby equivalently transforming the original problem into an unconstrained Lagrangian dual problem. Building on this duality result, the authors propose a hybrid solution framework that integrates dual ascent with offline value function approximation. This approach significantly reduces online computational complexity while preserving both optimality and probabilistic feasibility of the solution. Numerical experiments demonstrate that, compared to existing online model predictive control strategies, the proposed method achieves comparable control performance with substantially improved computational efficiency.