InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling

2025-07-23

AI Summary

This paper addresses the challenge of material estimation being susceptible to visible-light interference in large-scale, dynamic, relightable scene reconstruction from a single RGB+LiDAR sequence. We propose a physics-driven cross-modal inverse rendering framework. Our key contributions are: (1) a physically grounded LiDAR shading model that formulates intensity values as a wavelength-dependent reflectance coupled with sensor-specific response; (2) an RGB-LiDAR material consistency loss enabling joint optimization of color and reflectance properties; and (3) integration of differentiable rendering with cross-modal co-training. Evaluated on urban and indoor scenes, our method achieves high-fidelity novel-view synthesis, arbitrary relighting, nighttime simulation, and dynamic object insertion. It significantly outperforms state-of-the-art methods in both reconstruction fidelity and generalization capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present InvRGB+L, a novel inverse rendering model that reconstructs large, relightable, and dynamic scenes from a single RGB+LiDAR sequence. Conventional inverse graphics methods rely primarily on RGB observations and use LiDAR mainly for geometric information, often resulting in suboptimal material estimates due to visible light interference. We find that LiDAR's intensity values-captured with active illumination in a different spectral range-offer complementary cues for robust material estimation under variable lighting. Inspired by this, InvRGB+L leverages LiDAR intensity cues to overcome challenges inherent in RGB-centric inverse graphics through two key innovations: (1) a novel physics-based LiDAR shading model and (2) RGB-LiDAR material consistency losses. The model produces novel-view RGB and LiDAR renderings of urban and indoor scenes and supports relighting, night simulations, and dynamic object insertions, achieving results that surpass current state-of-the-art methods in both scene-level urban inverse rendering and LiDAR simulation.

Problem

• Reconstructs relightable scenes from RGB+LiDAR data
• Improves material estimation using LiDAR intensity cues
• Overcomes RGB-centric limitations with physics-based modeling

Innovation

• Physics-based LiDAR shading model
• RGB-LiDAR material consistency losses
• Unified color and LiDAR reflectance modeling