Learning Gaussian Structure: Intervention-Guided Density Control for Feed-Forward Driving Reconstruction

2026-08-11

AI Summary

Existing feedforward Gaussian reconstruction methods struggle to accumulate gradient-based optimization of scene density during training and lack explicit modeling of cross-temporal observations. To address these limitations, this work proposes the Learning Gaussian Structure (LGS) framework, which dynamically adjusts Gaussian structures through an intervention-guided density control mechanism and introduces a cross-time query module to explicitly aggregate multi-temporal features, thereby enhancing the reliability of attribute prediction. LGS is the first to incorporate an intervention-response-based strategy for adding or removing Gaussians, enabling structure adaptation during inference and overcoming the temporal modeling constraints inherent in feedforward approaches. Experiments on the Waymo Open Dataset and PandaSet demonstrate that the proposed method significantly outperforms existing techniques, achieving more accurate reconstruction of dynamic driving scenes.

Abstract

Feed-forward Gaussian reconstruction has recently emerged as an efficient approach for driving scene reconstruction. However, prevailing LiDAR-based methods preserve the initial correspondence between observed points and Gaussian primitives, treating the initialized primitive set as the final representation. Unlike optimization-based 3DGS, these methods cannot accumulate gradients during training to determine how the scenes representation should be densified. Meanwhile, the shared sparse backbone only fuses observations from different timestamps implicitly, without explicitly aggregating cross-time evidence for individual primitives. In this paper, we present Learning Gaussian Structure (LGS), a framework that enhances both Gaussian structure and primitive attributes. Our key observation is that changes in local gradient responses induced by a prune or add intervention reveal whether the corresponding structural adjustment benefits reconstruction. Based on this observation, our Gaussian Densify Policy learns a Densify Map comprising Prune and Addition Scores from controlled interventions, and directly adjusts the Gaussian structure during inference. We further develop a compact Cross-Time Point Query that explicitly retrieves and aggregates neighboring features from Gaussian primitives at other timestamps for reliable attribute prediction. Extensive experiments on the Waymo Open Dataset and PandaSet demonstrate that LGS consistently outperforms existing methods.

Problem

• Gaussian reconstruction
• density control
• intervention-guided learning
• cross-time aggregation
• driving scene reconstruction

Innovation

• Gaussian Structure Learning
• Intervention-Guided Densification
• Cross-Time Feature Aggregation
• Feed-Forward 3D Reconstruction
• LiDAR Scene Representation