Problem
Research questions and friction points this paper is trying to address.
loco-manipulation
reinforcement learning
sparse rewards
reward shaping
robot autonomy
Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
🤖 AI Summary
This work addresses the challenge of scaling reinforcement learning to complex loco-manipulation tasks, where conventional approaches rely heavily on handcrafted dense rewards. The authors propose a novel framework that leverages sample-based model predictive control (SMPC) as an automated expert policy generator to efficiently construct large-scale offline datasets in simulation. This dataset is then used within a hierarchical architecture combining offline-to-online reinforcement learning under sparse rewards with a low-level dynamically stable controller. Notably, the method eliminates the need for manual reward engineering and enables agents trained solely with sparse rewards to surpass the performance of the SMPC teacher policy. The approach demonstrates strong empirical results, successfully deploying on both the Spot quadruped and G1 humanoid robots with high performance, robustness, and effective sim-to-real transfer.
📝 Abstract
Integrating locomotion and manipulation is essential for robot autonomy, but scaling standard Reinforcement Learning (RL) to complex tasks is severely bottlenecked by the slow, manual process of dense reward shaping. To bypass this limitation, we leverage Sample-based Model Predictive Control (SMPC) entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets. Because this data solves the fundamental exploration problem, we can train an off-policy RL agent using purely sparse task rewards, drastically reducing the time required to learn new skills and eliminating the need for manual tuning. Integrating this high-level agent with a low-level dynamic stability controller yields more optimal behaviors that strictly align with true task objectives, ultimately allowing the learned policies to surpass the original optimal control teacher. We validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies, including an arm-equipped Spot quadruped and a G1 humanoid.
Innovation
Methods, ideas, or system contributions that make the work stand out.
loco-manipulation
sparse reward RL
SMPC demonstrations
offline-to-online RL
sim-to-real transfer
🔎 Similar Papers
No similar papers found.
