ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos

2026-02-05

AI Summary

This work addresses the challenge of 3D reconstruction from monocular hand-object interaction videos, where severe occlusions and complex object motions hinder accurate recovery. To this end, we propose a feedforward neural network that, for the first time, jointly optimizes 2D mask inpainting and 3D shape completion within a unified framework, enabling high-fidelity reconstruction in under one minute without any preprocessing. To facilitate this task, we introduce the first large-scale, high-fidelity synthetic dataset of hand-object interactions. Extensive experiments demonstrate that our method achieves state-of-the-art reconstruction accuracy while offering a roughly 100× speedup in inference time compared to conventional optimization-based approaches.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Learning with Manifolds

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: https://github.com/Tao-11-chen/ForeHOI.

Problem

• 3D object reconstruction
• hand-object interaction
• monocular video
• occlusion
• embodied intelligence

Innovation

• feed-forward reconstruction
• hand-object interaction
• 3D object reconstruction
• occlusion handling
• synthetic dataset