High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians

2026-06-14

AI Summary

This work addresses the challenge of robustly reconstructing high-fidelity 4D hand–object interactions under severe occlusion, where existing methods often rely on object templates or physical markers and are sensitive to initial pose estimates. To overcome these limitations, we propose a marker- and template-free multi-view 4D reconstruction framework. Our approach first leverages a multi-view spatio-temporal Transformer to fuse geometric and temporal cues across views, yielding a reliable initialization. Subsequently, we introduce a physics-aware 3D Gaussian optimization mechanism that incorporates tetrahedral deformation constraints, collision detection, and appearance disentanglement to achieve fine-grained reconstruction. Extensive experiments on both public and in-house datasets demonstrate that our method produces robust, artifact-free, high-fidelity results, offering an efficient solution for automated 4D asset generation.

Abstract

The growing demand for high-fidelity 4D hand-object interaction (HOI) data in embodied AI and spatial computing is currently bottlenecked by the reliance on pre-scanned object templates and physical markers. While recent methods have demonstrated promising results in reconstructing 4D hand-object interaction from videos, they are highly sensitive to initial estimates of hand and object poses. Yet, estimating these poses from images is challenging, in particular under severe occlusion which is inherent in hand-object interaction scenarios. We propose a novel system for the robust and accurate reconstruction of hands and objects from synchronized and calibrated multi-view videos without requiring any templates or markers. Our system consists of two main components with key innovations: (1) a multi-view feed-forward transformer model that aggregates cross-view geometry and temporal cues to provide a reliable, metric-consistent initialization for both poses and dense object geometry, and (2) a hand-object physics-aware Gaussian-based optimization framework to refine the initial estimates, integrating tetrahedral constraints, collision refinement, and appearance decomposition to produce physically plausible and visually accurate reconstruction. Validated on public benchmarks and an extensive internal dataset, our pipeline achieves highly robust, artifact-free reconstruction, providing an efficient foundation for automated 4D asset generation. Our project page are available at https://zyshen021.github.io/HOSTPG/.

Problem

• 4D hand-object interaction
• occlusion
• pose estimation
• template-free reconstruction
• markerless capture

Innovation

• multi-view spatiotemporal tracking
• physics-aware Gaussians
• template-free reconstruction
• hand-object interaction
• 4D capture