DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

2026-04-22

AI Summary

This work proposes DeVI, a novel framework that enables zero-shot dexterous manipulation solely from text-guided 2D synthetic videos, eliminating the need for high-quality 3D motion demonstrations. By integrating 3D hand pose estimation with robust 2D object tracking, DeVI constructs a hybrid tracking reward to guide reinforcement learning policies in imitating text-specified actions. The approach supports multi-object scenarios and diverse text-driven behaviors without requiring real-world 3D demonstrations. Experimental results demonstrate that DeVI significantly outperforms existing 3D-demonstration-based methods on dexterous hand–object interaction tasks and exhibits strong generalization capabilities to unseen objects and varied natural language instructions.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advances in video generative models enable the synthesis of realistic human-object interaction videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control. We present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects. To overcome the imprecision of generative 2D cues, we introduce a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking. Unlike methods relying on high-quality 3D kinematic demonstrations, DeVI requires only the generated video, enabling zero-shot generalization across diverse objects and interaction types. Extensive experiments demonstrate that DeVI outperforms existing approaches that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions. We further validate the effectiveness of DeVI in multi-object scenes and text-driven action diversity, showcasing the advantage of using video as an HOI-aware motion planner.

Problem

• dexterous manipulation
• human-object interaction
• synthetic video
• physics-based control
• imitation learning

Innovation

• dexterous manipulation
• video imitation
• physics-based control
• synthetic video
• zero-shot generalization