HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos

2025-01-06

AI Summary

Reconstructing hand trajectories in world coordinates from egocentric videos is challenging due to coupled hand–camera motion. This paper proposes a decoupled modeling framework that separately estimates camera trajectory and hand-relative motion. Methodologically, we design an adaptive egocentric SLAM module for robust camera localization and integrate a temporal hand-motion interpolation network to mitigate degradation caused by occlusions and rapid motion. Furthermore, we introduce multi-stage joint optimization with world-coordinate alignment constraints to ensure global consistency. Evaluated on multiple egocentric benchmarks, our approach achieves state-of-the-art performance: it reduces hand world-trajectory error by 32% and absolute trajectory error (ATE) of the camera by 41%. Notably, it enables accurate recovery of hand trajectories even after brief out-of-view periods.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Despite the advent in 3D hand pose estimation, current methods predominantly focus on single-image 3D hand reconstruction in the camera frame, overlooking the world-space motion of the hands. Such limitation prohibits their direct use in egocentric video settings, where hands and camera are continuously in motion. In this work, we propose HaWoR, a high-fidelity method for hand motion reconstruction in world coordinates from egocentric videos. We propose to decouple the task by reconstructing the hand motion in the camera space and estimating the camera trajectory in the world coordinate system. To achieve precise camera trajectory estimation, we propose an adaptive egocentric SLAM framework that addresses the shortcomings of traditional SLAM methods, providing robust performance under challenging camera dynamics. To ensure robust hand motion trajectories, even when the hands move out of view frustum, we devise a novel motion infiller network that effectively completes the missing frames of the sequence. Through extensive quantitative and qualitative evaluations, we demonstrate that HaWoR achieves state-of-the-art performance on both hand motion reconstruction and world-frame camera trajectory estimation under different egocentric benchmark datasets. Code and models are available on https://hawor-project.github.io/ .

Problem

• Hand Pose Estimation
• First-person View
• Simultaneous Hand and Camera Motion

Innovation

• First-person View Hand Motion Recovery
• Improved SLAM System
• Specialized Network for Hand Occlusion Prediction