LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans

2025-07-03

AI Summary

This paper addresses the challenge of reconstructing compact, photorealistic, and interactive 3D virtual scenes from RGB-D indoor scans. Methodologically: (1) it employs a structured scene graph for semantic understanding and object-level segmentation; (2) introduces an unsupervised object retrieval module achieving state-of-the-art cross-modal similarity on Scan2CAD; (3) proposes a robust material rendering module enabling spatially varying PBR material recovery under occlusion, misalignment, and low-light conditions; and (4) integrates high-fidelity 3D artistic assets with a physics simulation engine to ensure object independence, articulated joint connectivity, and physically grounded interaction. The resulting scenes are lightweight, editable, and compatible with standard graphics pipelines. Extensive evaluation on real-world scans and public benchmarks demonstrates high visual fidelity and strong interactive capabilities, validating applicability in AR/VR, robotics, and digital twin systems.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c

Problem

• Converts RGB-D scans into interactive 3D virtual replicas
• Enhances realism with high-quality materials and physical properties
• Achieves state-of-the-art similarity in object retrieval and material painting

Innovation

• Converts RGB-D scans into interactive 3D replicas
• Retrieves artist-crafted models for scene reconstruction
• Enhances realism with highquality material painting