The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation

2025-04-11

AI Summary

Current methods for hand motion prediction in first-person videos are limited to visible hand locations, neglecting joint-level dynamics and handling of occlusions or out-of-view scenarios. To address these limitations, this paper proposes EgoH4—the first end-to-end diffusion-based Transformer framework enabling long-horizon, joint prediction of 3D hand trajectories and poses. Methodologically, EgoH4 is the first to explicitly model invisible hand motion by integrating full-body pose priors and hand visibility estimation; it introduces a 3D-to-2D reprojection loss and multi-task joint denoising (for hands, body, and visibility). Evaluated on the Ego-Exo4D dataset, EgoH4 achieves state-of-the-art performance: it reduces average displacement error (ADE) for hand trajectories by 3.4 cm and mean per-joint position error (MPJPE) for hand poses by 5.1 cm over existing baselines—demonstrating significant improvements in both accuracy and robustness to occlusion and out-of-frame motion.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: State Estimation

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Forecasting hand motion and pose from an egocentric perspective is essential for understanding human intention. However, existing methods focus solely on predicting positions without considering articulation, and only when the hands are visible in the field of view. This limitation overlooks the fact that approximate hand positions can still be inferred even when they are outside the camera's view. In this paper, we propose a method to forecast the 3D trajectories and poses of both hands from an egocentric video, both in and out of the field of view. We propose a diffusion-based transformer architecture for Egocentric Hand Forecasting, EgoH4, which takes as input the observation sequence and camera poses, then predicts future 3D motion and poses for both hands of the camera wearer. We leverage full-body pose information, allowing other joints to provide constraints on hand motion. We denoise the hand and body joints along with a visibility predictor for hand joints and a 3D-to-2D reprojection loss that minimizes the error when hands are in-view. We evaluate EgoH4 on the Ego-Exo4D dataset, combining subsets with body and hand annotations. We train on 156K sequences and evaluate on 34K sequences, respectively. EgoH4 improves the performance by 3.4cm and 5.1cm over the baseline in terms of ADE for hand trajectory forecasting and MPJPE for hand pose forecasting. Project page: https://masashi-hatano.github.io/EgoH4/

Problem

• Forecasting 3D hand motion from egocentric videos
• Predicting hand poses even when hands are out of view
• Improving accuracy over existing hand trajectory methods

Innovation

• Diffusion-based transformer for hand forecasting
• Leverages full-body pose constraints
• Uses 3D-to-2D reprojection loss