Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

2026-06-29

AI Summary

This work addresses the challenge of 3D hand–object pose estimation from in-the-wild first-person RGB videos, where severe occlusions and motion blur during contact hinder performance, compounded by poor generalization of existing methods and scarcity of annotated supervision. To tackle these issues, the authors propose HOPformer, an end-to-end Transformer model that leverages hand priors to guide object feature learning and jointly predicts 3D poses of both hands and the interacting object in a single forward pass. Key contributions include the introduction of EPIC-Contact—the first large-scale in-the-wild hand–object contact dataset—augmented with dense bidirectional contact correspondence annotations, and a novel cross-attention-based architecture for joint modeling. Experiments demonstrate that HOPformer achieves a 6.2 percentage point improvement over prior art on ARCTIC, reaching 82.4% success rate, and nearly doubles performance on EPIC-Contact while reducing contact error by 75%.

Abstract

Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.

Problem

• egocentric vision
• 3D hand-object pose estimation
• in-the-wild
• occlusion
• contact ambiguity

Innovation

• egocentric vision
• 3D hand-object pose estimation
• contact correspondence
• transformer architecture
• in-the-wild dataset