LAMP: Latent Motion Prior-Guided Real-World Learning for Dexterous Hand Manipulation

2026-07-07

AI Summary

This work addresses the challenge of efficiently and safely learning dexterous manipulation in real-world settings, where high-dimensional hand action spaces are prone to error amplification from imperfect demonstrations and disruptive exploration. The authors propose a three-stage framework: first, a history-conditioned latent motion prior is pretrained from a small set of demonstrations to compress high-dimensional hand actions into a low-dimensional latent space; second, a visuomotor policy is trained to predict offsets to both the robot arm’s native commands and the latent hand actions; and third, residual reinforcement learning is performed within the shared latent space to locally refine the policy. This approach enables efficient and safe online exploration while preserving contact consistency. Evaluated on four real-world dexterous manipulation tasks, the method achieves an average imitation learning success rate of 56.25%, which improves to 98.75% after online reinforcement learning—with three tasks reaching 100% success.

Abstract

Real-world learning for dexterous hands remains brittle because high-dimensional hand actions amplify imitation errors and make reinforcement-learning exploration prone to contact-breaking motion. While combining imitation learning (IL) with online reinforcement learning (RL) can reduce manual supervision, unconstrained exploration in raw hand-action spaces is sample-inefficient and risky for physical hardware. We introduce a latent motion prior module (\prior{}) that maps recent hand-action histories to a compact, history-conditioned latent prior and decodes continuous latent commands into executable high-dimensional hand targets. Built on this prior, \method{} is a three-stage real-world dexterous learning framework: it pretrains \prior{} from demonstrations, trains a visuomotor policy that predicts native arm commands and latent hand-action offsets, and improves the policy with online residual RL in the same latent hand-action space. This shared, decodable interface lets residual exploration make local corrections near demonstrated, contact-consistent hand motions rather than perturbing every finger joint independently. We evaluate \method{} on four real-robot dexterous manipulation tasks against raw, linear, and discrete hand-action interfaces. Starting from small task-specific demonstration sets, \method{} achieves a 56.25\% average IL success rate and raises it to 98.75\% after online RL, reaching 100\% final success on three tasks and 95\% on the remaining task.

Problem

• dexterous hand manipulation
• imitation learning
• reinforcement learning
• real-world learning
• high-dimensional action space

Innovation

• latent motion prior
• dexterous hand manipulation
• imitation learning
• residual reinforcement learning
• real-world robot learning