Constrained Sampling to Guide Universal Manipulation RL

2026-02-09

AI Summary

This work addresses the challenge of effective exploration in contact-rich, sparsely rewarded environments where general-purpose reinforcement learning struggles to discover complex manipulation strategies. The authors propose a Sample-Guided RL framework that leverages a differentiable model-based solver—incorporating collision, contact, and force constraints—to construct a low-dimensional manifold of feasible states and guide policy learning through targeted sampling. By integrating black-box optimization to generate open-loop trajectories and introducing state-visit bias alongside behavioral cloning loss, the method significantly enhances both training efficiency and performance of goal-conditioned policies. Evaluated on a simplified two-sphere environment and Panda robot arm tasks, the approach substantially outperforms baseline methods, achieving high success rates in reaching statically stable states and demonstrating diverse whole-body contact-aware manipulation strategies.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We consider how model-based solvers can be leveraged to guide training of a universal policy to control from any feasible start state to any feasible goal in a contact-rich manipulation setting. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies, especially in sparse-reward settings. Our approach is based on the idea of a lower-dimensional manifold of feasible, likely-visited states during such manipulation and to guide RL with a sampler from this manifold. We propose Sample-Guided RL, which uses model-based constraint solvers to efficiently sample feasible configurations (satisfying differentiable collision, contact, and force constraints) and leverage them to guide RL for universal (goal-conditioned) manipulation policies. We study using this data directly to bias state visitation, as well as using black-box optimization of open-loop trajectories between random configurations to impose a state bias and optionally add a behavior cloning loss. In a minimalistic double sphere manipulation setting, Sample-Guided RL discovers complex manipulation strategies and achieves high success rates in reaching any statically stable state. In a more challenging panda arm setting, our approach achieves a significant success rate over a near-zero baseline, and demonstrates a breadth of complex whole-body-contact manipulation strategies.

Problem

• constrained sampling
• universal manipulation
• reinforcement learning
• contact-rich manipulation
• sparse-reward

Innovation

• Sample-Guided RL
• constraint-based sampling
• universal manipulation policy
• feasible state manifold
• model-based guidance