Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control

2025-10-14

AI Summary

To address the limited robustness of Model Predictive Control (MPC) under model mismatch and real-time constraints, and the poor interpretability and generalization of end-to-end Reinforcement Learning (RL), this paper proposes a residual MPC framework. At the torque level, it integrates physics-based MPC with a differentiable, end-to-end RL policy, where the model prior guides policy learning. GPU-accelerated parallel dynamics optimization and large-scale parallel simulation enable high-frequency closed-loop control. The method achieves zero-shot transfer to unseen gaits and non-flat terrains, significantly improving sample efficiency and command-tracking capability—extending coverage across a broader velocity range. Compared to standalone MPC or end-to-end RL, our approach preserves interpretability while enhancing out-of-distribution robustness and convergence performance.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Model Predictive Control (MPC) provides interpretable, tunable locomotion controllers grounded in physical models, but its robustness depends on frequent replanning and is limited by model mismatch and real-time computational constraints. Reinforcement Learning (RL), by contrast, can produce highly robust behaviors through stochastic training but often lacks interpretability, suffers from out-of-distribution failures, and requires intensive reward engineering. This work presents a GPU-parallelized residual architecture that tightly integrates MPC and RL by blending their outputs at the torque-control level. We develop a kinodynamic whole-body MPC formulation evaluated across thousands of agents in parallel at 100 Hz for RL training. The residual policy learns to make targeted corrections to the MPC outputs, combining the interpretability and constraint handling of model-based control with the adaptability of RL. The model-based control prior acts as a strong bias, initializing and guiding the policy towards desirable behavior with a simple set of rewards. Compared to standalone MPC or end-to-end RL, our approach achieves higher sample efficiency, converges to greater asymptotic rewards, expands the range of trackable velocity commands, and enables zero-shot adaptation to unseen gaits and uneven terrain.

Problem

• Blending MPC and RL to overcome model mismatch and computational constraints
• Combining interpretable model-based control with adaptable learning-based corrections
• Enabling robust locomotion adaptation to unseen terrains and velocity commands

Innovation

• Blends MPC and RL outputs at torque level
• Uses GPU-parallelized kinodynamic MPC formulation
• Learns residual corrections to MPC via RL