Towards Egocentric 3D Hand Pose Estimation in Unseen Domains

2026-01-10

arXiv.org

AI Summary

This work addresses the poor generalization of first-person 3D hand pose estimation in unseen domains, primarily caused by scarce training data and depth bias induced by variations in camera intrinsic parameters. To mitigate this, the authors propose an intrinsics-invariant depth representation that predicts keypoint z-coordinates in a normalized virtual camera space, thereby enabling camera-agnostic depth estimation. Furthermore, they introduce a test-time self-supervised optimization mechanism that leverages 3D consistency loss to dynamically adapt to target domain characteristics. The proposed approach substantially improves cross-domain generalization, reducing average pose errors by 71% on H2O and 41% on AssemblyHands. Notably, it achieves these gains using only 1/3.5 to 1/14 of the training data required by existing single-stage methods and approaches the performance of two-stage models.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We present V-HPOT, a novel approach for improving the cross-domain performance of 3D hand pose estimation from egocentric images across diverse, unseen domains. State-of-the-art methods demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception -- overfitting to specific camera intrinsics. Our method addresses this by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model's depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing approximately x3.5 to x14 less data.

Problem

• egocentric
• 3D hand pose estimation
• cross-domain
• unseen domains
• camera intrinsics

Innovation

• cross-domain generalization
• camera-invariant depth estimation
• self-supervised test-time optimization
• egocentric 3D hand pose
• virtual camera space