MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

2026-06-24

AI Summary

Reinforcement learning in locomotion control often converges to undesirable local optima—such as jittering or sliding—that are impractical for real-world deployment, making it challenging to generate functional gaits. This work proposes a novel approach that injects trajectory samples generated by a model predictive controller (MPC) solving the same Markov decision process into the experience replay buffer. By reshaping the replay distribution, the method steers policy learning toward designer-preferred, high-quality gaits without altering the reward function or resorting to adversarial imitation learning. Evaluated in both 2D simulation and on the Go2 quadruped robot, the approach achieves gait performance comparable to methods relying on either 21 carefully engineered reward terms or adversarial priors, using only one or two simple reward components. To our knowledge, this is the first demonstration of transferring controller preferences solely through replay distribution modulation.

Abstract

Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.

Problem

• reinforcement learning
• locomotion
• behavior bias
• undeployable gaits
• off-policy RL

Innovation

• MPC-Injection
• off-policy reinforcement learning
• model predictive control
• locomotion
• replay buffer biasing