VEPHand: View-Efficient Photometric Hand Performance Capture at Scale

2026-06-14

AI Summary

This work addresses the challenges of dynamic 3D hand reconstruction in real-world multi-view settings with limited viewpoints (approximately 20) and no foreground masks, where geometric ambiguity, background clutter, and self-contact pose significant difficulties. The authors propose an end-to-end, efficient framework for hand motion capture and registration: first, a mask-free neural scene representation combined with density regularization robustly recovers high-fidelity hand geometry; second, a physics-based registration module optimizes intrinsic offsets of a personalized hand template within a tetrahedral mesh to accurately model nonlinear skin deformations and self-contact. Evaluated on a large-scale dataset comprising over 12,000 sequences, the method achieves state-of-the-art performance in both reconstruction fidelity and registration accuracy under mask-free, visually efficient conditions.

Abstract

Robust, high-fidelity 3D hand capture, while fundamental to digital human creation, remains challenging with practical multi-view systems that balance rich photometry with the geometric ambiguities of reconstruction arising from limited viewpoint density. This paper presents an end-to-end pipeline for dynamic hand performance capture and registration, specifically designed for view-efficient setups ($\sim$20 views). We address key challenges with two primary innovations. First, to overcome reconstruction difficulties like limited view overlap and background clutter, our mask-free neural method robustly extracts detailed hand geometry and appearance from unmasked images using scene parameterization and scenario-specific density regularization. Second, addressing registration challenges such as accurately capturing non-linear skin deformations and ensuring plausible results during severe self-contact, we propose a physics-inspired framework. It aligns reconstructions to a personalized hand model by optimizing intrinsic volumetric offsets within its canonical tetrahedral mesh, alongside pose parameters. This approach, supported by robust losses and optimization, captures fine surface deformations, ensures plausible results under severe articulation and self-contact, and demonstrates strong tolerance to input noise. We demonstrate the scalability and robustness of our automated pipeline on an extensive dataset of over 12,000 sequences, from which we also derive a large-scale, high-quality synthetic 2D/3D hand dataset for training downstream tasks. This showcases its effectiveness for single hands, intricate two-hand interactions, and natural hand-object manipulations. Our method achieves state-of-the-art reconstruction fidelity in view-efficient, unmasked scenarios and highly accurate registration. Our project page are available at https://zyshen021.github.io/VEPHand/.

Problem

• hand performance capture
• view-efficient
• photometric reconstruction
• 3D hand registration
• self-contact

Innovation

• view-efficient
• mask-free neural reconstruction
• physics-inspired registration
• volumetric offset optimization
• hand performance capture