HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control

2026-07-02

AI Summary

Existing approaches rely on multi-view setups or instrumented motion capture, making controllable hand generation in unconstrained egocentric videos challenging. This work proposes the first protagonist-centric 3D hand annotation pipeline tailored for in-the-wild monocular egocentric videos, yielding the high-quality dataset EgoVid-Pro. To decouple camera and hand motion at the representation level, we introduce the Plücker Hand Map formulation. Integrating monocular 3D reconstruction, semantic-geometric joint filtering, and Plücker-ray augmentation, our method significantly outperforms prior art in both reconstruction fidelity and control accuracy, demonstrating robust generalization across diverse everyday scenarios.

Abstract

We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video. Such generality is bottlenecked by the scarcity of scalable 3D hand annotations: large egocentric corpora lack finger-level labels, whereas precise hand datasets are confined to narrow, instrumented settings, limiting prior hand-controlled generators to restricted scene distributions. We instead annotate 3D hands directly on in-the-wild egocentric video through monocular reconstruction, introducing a protagonist-centered annotation pipeline that filters the reconstructions at the action-semantic, image-quality, and 3D-geometric levels to build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes. To resolve the camera-hand entanglement induced by large ego-motion, we further propose the Plücker Hand Map, a 3D-aware control signal that extends Plücker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level. Experiments show that \method surpasses prior hand-controlled generators in reconstruction fidelity and control accuracy, and generalizes to out-of-distribution everyday scenes beyond the laboratory datasets on which prior methods rely.

Problem

• egocentric video generation
• hand control
• 3D hand annotation
• camera-hand entanglement
• in-the-wild data

Innovation

• egocentric video generation
• hand motion control
• monocular 3D hand reconstruction
• camera-hand disentanglement
• Plücker Hand Map