DRAWER: Digital Reconstruction and Articulation With Environment Realism

2025-04-21

AI Summary

This work addresses the challenge of reconstructing monocular indoor videos into high-fidelity, interactive digital twin environments—where static scene representations hinder physics-based simulation and real-time interaction. To this end, we propose a dual-scene representation framework that jointly models geometry and appearance via a hybrid neural radiance field (NeRF) and explicit mesh representation. We further introduce a novel semantic-geometric articulation module for movable parts, which detects hinges and reconstructs physically simulatable shapes to enable articulated motion simulation. Our method is end-to-end integrated with neural rendering and deployed within Unreal Engine and robotics simulation platforms. Experiments demonstrate a 32% reduction in geometric reconstruction error and interactive response latency under 16 ms. To our knowledge, this is the first approach to achieve a closed-loop pipeline from real-world capture → simulation training → real-world deployment, successfully generating game-grade interactive digital environments.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Creating virtual digital replicas from real-world data unlocks significant potential across domains like gaming and robotics. In this paper, we present DRAWER, a novel framework that converts a video of a static indoor scene into a photorealistic and interactive digital environment. Our approach centers on two main contributions: (i) a reconstruction module based on a dual scene representation that reconstructs the scene with fine-grained geometric details, and (ii) an articulation module that identifies articulation types and hinge positions, reconstructs simulatable shapes and appearances and integrates them into the scene. The resulting virtual environment is photorealistic, interactive, and runs in real time, with compatibility for game engines and robotic simulation platforms. We demonstrate the potential of DRAWER by using it to automatically create an interactive game in Unreal Engine and to enable real-to-sim-to-real transfer for robotics applications.

Problem

• Converts video of static indoor scenes into interactive digital environments
• Reconstructs scenes with fine-grained geometric details and simulatable components
• Enables real-time photorealistic environments for gaming and robotics simulations

Innovation

• Dual scene representation for detailed reconstruction
• Articulation module for simulatable shapes integration
• Real-time photorealistic interactive virtual environments