V2P-Manip: Learning Dexterous Manipulation from Monocular Human Videos

2026-06-15

AI Summary

This work addresses the challenge of efficiently extracting dexterous manipulation trajectories from monocular videos of human demonstrations that simultaneously achieve high visual fidelity and physical plausibility, thereby serving as a cost-effective alternative to expensive teleoperation data. To this end, the authors propose an end-to-end framework that integrates 3D hand reconstruction, trajectory optimization, and reinforcement learning, featuring a novel two-stage refinement mechanism that jointly leverages visual perception and physical constraints to ensure spatial alignment and dynamic consistency. The resulting method generates hand-agnostic manipulation priors that generalize across diverse hand morphologies, achieving over 75% average success rate across multiple tasks on the TACO and OakInk benchmarks—significantly outperforming existing approaches—and demonstrates strong adaptability and sample efficiency in unstructured environments.

Abstract

Achieving autonomous robotic dexterous manipulation requires precise, human-like action sequences at scale. As a scalable supplement to costly teleoperation data, extracting trajectories with both visual fidelity and physical plausibility from monocular videos represents a promising frontier in embodied AI. To this end, we introduce V2P-Manip, an efficient framework designed to learn dexterous manipulation policies directly from human demonstration videos. We establish an efficient, integrated pipeline encompassing 3D asset acquisition, trajectory estimation, and dexterous policy learning. To bridge the gap between visual perception and physical constraints, we introduce a two-stage refinement process to enforce spatial alignment and physical consistency. Evaluations on the TACO and OakInk benchmarks demonstrate that our approach significantly outperforms previous methods in pose accuracy, adaptability to unstructured environments, and training efficiency. Ultimately, experimental results confirm an average success rate of over 75% across multiple synthetic manipulation tasks and validate the adaptability of the extracted manipulation priors across diverse dexterous hand embodiments.

Problem

• dexterous manipulation
• monocular videos
• trajectory extraction
• embodied AI
• physical plausibility

Innovation

• dexterous manipulation
• monocular video
• trajectory estimation
• physical consistency
• embodied AI