The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

2026-06-29

AI Summary

This work addresses the challenge of 4D hand motion reconstruction from egocentric videos, where existing methods often fail due to occlusions or rely on sparsely annotated temporal modules, limiting their ability to model dynamics, occlusions, and hand–object interactions. To overcome these limitations, we introduce a novel approach that leverages large-scale pretrained video diffusion models for the first time in this domain. We propose a rendering-guided feature adaptation mechanism that fine-tunes model features via hand-overlay rendering, enabling the model to focus on hand-specific cues while preserving general visual priors. An end-to-end decoder is further designed to directly recover metric-scale 4D hand poses from full video frames—without requiring detectors, inpainting modules, or test-time optimization. Our method significantly outperforms state-of-the-art approaches on ARCTIC, HOT3D, and HOI4D, demonstrating the efficacy of video diffusion models as a new paradigm for hand reconstruction and offering a practical pathway for in-the-wild data collection in embodied intelligence.

Abstract

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.

Problem

• 4D hand motion reconstruction
• egocentric video
• occlusion
• hand-object interaction
• video-based methods

Innovation

• video diffusion models
• 4D hand motion reconstruction
• egocentric vision
• hand-object interaction
• world priors