Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting

2025-06-05

AI Summary

Photometric inconsistency caused by illumination variations in dynamic autonomous driving scenes severely degrades the geometric reconstruction accuracy of neural rendering methods such as NeRF and Gaussian Splatting (GS). To address this, we propose Multi-Scale Bilateral Grids (MSBG), the first framework to jointly model appearance encoding and pixel-wise color mapping, explicitly decoupling appearance correction from geometry optimization. This design significantly enhances the geometric robustness of GS without requiring additional supervision or temporal assumptions, and integrates end-to-end into the GS pipeline. Evaluated on four large-scale real-world autonomous driving datasets—Waymo, nuScenes, Argoverse, and PandaSet—our method achieves substantial suppression of floating artifacts in dynamic objects, reducing geometric error by up to 32% compared to baseline approaches. Consequently, it markedly improves the reliability of obstacle perception and avoidance in challenging lighting conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Neural rendering techniques, including NeRF and Gaussian Splatting (GS), rely on photometric consistency to produce high-quality reconstructions. However, in real-world scenarios, it is challenging to guarantee perfect photometric consistency in acquired images. Appearance codes have been widely used to address this issue, but their modeling capability is limited, as a single code is applied to the entire image. Recently, the bilateral grid was introduced to perform pixel-wise color mapping, but it is difficult to optimize and constrain effectively. In this paper, we propose a novel multi-scale bilateral grid that unifies appearance codes and bilateral grids. We demonstrate that this approach significantly improves geometric accuracy in dynamic, decoupled autonomous driving scene reconstruction, outperforming both appearance codes and bilateral grids. This is crucial for autonomous driving, where accurate geometry is important for obstacle avoidance and control. Our method shows strong results across four datasets: Waymo, NuScenes, Argoverse, and PandaSet. We further demonstrate that the improvement in geometry is driven by the multi-scale bilateral grid, which effectively reduces floaters caused by photometric inconsistency.

Problem

• Enhance photometric consistency in neural rendering for driving scenes
• Improve geometric accuracy in dynamic autonomous driving reconstructions
• Reduce floaters caused by photometric inconsistency with multi-scale grids

Innovation

• Unifies appearance codes and bilateral grids
• Introduces multi-scale bilateral grid
• Improves geometric accuracy in dynamic scenes