TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes

2025-01-07

AI Summary

This work addresses the challenge of high-fidelity 3D object texture reconstruction from monocular video in hand–object interaction scenes, where occlusion by hands, indirect illumination, and pose estimation errors cause severe texture distortion. We propose the first end-to-end framework for photorealistic texture reconstruction. Methodologically, we explicitly model hand-induced occlusion to estimate object visibility, and jointly optimize skin reflectance properties and environment lighting to disentangle hand–object physical interactions. Our approach integrates composite radiance field rendering, differentiable physically based rendering, and multi-stage geometry–texture co-optimization, enabling joint hand–object pose estimation. Experiments demonstrate significant improvements over state-of-the-art methods in albedo, shadow, and specular region recovery accuracy. To our knowledge, this is the first method achieving robust, photorealistic texture reconstruction from a single viewpoint in dynamic hand–object interaction scenarios.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Interaction Techniques and Devices

Application Category

• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Reconstructing 3D models of dynamic, real-world objects with high-fidelity textures from monocular frame sequences has been a challenging problem in recent years. This difficulty stems from factors such as shadows, indirect illumination, and inaccurate object-pose estimations due to occluding hand-object interactions. To address these challenges, we propose a novel approach that predicts the hand's impact on environmental visibility and indirect illumination on the object's surface albedo. Our method first learns the geometry and low-fidelity texture of the object, hand, and background through composite rendering of radiance fields. Simultaneously, we optimize the hand and object poses to achieve accurate object-pose estimations. We then refine physics-based rendering parameters - including roughness, specularity, albedo, hand visibility, skin color reflections, and environmental illumination - to produce precise albedo, and accurate hand illumination and shadow regions. Our approach surpasses state-of-the-art methods in texture reconstruction and, to the best of our knowledge, is the first to account for hand-object interactions in object texture reconstruction.

Problem

• 3D Reconstruction
• Single View
• Pose Estimation

Innovation

• 3D Reconstruction
• Hand-Object Interaction
• Texture Rendering