FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos

2026-04-07

AI Summary

This work proposes a method for automatically reconstructing interactive indoor scene digital twins from in-the-wild, first-person RGB-D interaction videos. Without requiring controlled environments, multi-state data collection, or CAD priors, the approach jointly discovers articulated parts, estimates kinematic parameters, tracks 3D motion, and reconstructs geometry in a canonical space to produce simulation-compatible mesh models. Evaluated on newly established real-world and simulated benchmarks, the method substantially outperforms existing approaches, achieving up to a 50-point improvement in part segmentation mIoU, reducing joint and pose errors by 5–10×, and significantly enhancing reconstruction accuracy. The resulting models support direct export to URDF/USD formats, enabling immediate use in robotic interaction and simulation pipelines.

Abstract

We present FunRec, a method for reconstructing functional 3D digital twins of indoor scenes directly from egocentric RGB-D interaction videos. Unlike existing methods on articulated reconstruction, which rely on controlled setups, multi-state captures, or CAD priors, FunRec operates directly on in-the-wild human interaction sequences to recover interactable 3D scenes. It automatically discovers articulated parts, estimates their kinematic parameters, tracks their 3D motion, and reconstructs static and moving geometry in canonical space, yielding simulation-compatible meshes. Across new real and simulated benchmarks, FunRec surpasses prior work by a large margin, achieving up to +50 mIoU improvement in part segmentation, 5-10 times lower articulation and pose errors, and significantly higher reconstruction accuracy. We further demonstrate applications on URDF/USD export for simulation, hand-guided affordance mapping and robot-scene interaction.

Problem

• functional 3D reconstruction
• egocentric interaction videos
• articulated objects
• digital twins
• simulation-compatible scenes

Innovation

• functional 3D reconstruction
• egocentric interaction video
• articulated object modeling
• simulation-compatible digital twin
• kinematic parameter estimation