Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control

2026-09-17

AI Summary

为解决视觉策略学习中的高计算成本和局部优化问题，提出采样引导策略搜索(SGPS)，结合基于采样的模型预测控制与一阶策略优化。

Abstract

Learning visual policies for locomotion and manipulation requires coordinating contact with the environment and can incur substantial computation and GPU memory costs. First-order policy gradients (FoPG) reduce training cost through differentiable simulation, but local optimization can converge to unintended contact patterns. To address this shortfall, we propose Sampling-Guided Policy Search (SGPS), which couples recurring action-target refinement by sampling-based model-predictive control with first-order policy optimization. Behavior cloning initializes the policy from sampled actions; training then alternates sampling-based refinement with short-horizon FoPG updates under perturbed initial states and randomized dynamics. For visual policy training, we use a decoupled FoPG formulation that excludes rendering from the computation graph, enabling direct learning from depth observations without a state-policy teacher. On a single GPU, SGPS learns policies for locomotion, obstacle traversal, crate pushing, and bimanual carrying on simulated Unitree Go2 and G1 robots. Our experiments further show that refinement improves policy learning beyond initialization and tracking alone. For hardware deployment, the distilled policy transfers zero-shot to a real Go2 and uses onboard depth to autonomously trot, crawl, clear hurdles, and switch between these behaviors.

Problem

• visual policies
• first-order policy gradients
• contact patterns
• computation cost
• GPU memory

Innovation

• Sampling-Guided Policy Search (SGPS)
• model-predictive control
• first-order policy gradients
• decoupled FoPG formulation
• visual policies