VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction

2025-06-05

AI Summary

To address performance bottlenecks in monocular 3D semantic occupancy and scene flow joint prediction—arising from occlusion and imbalanced dynamic object distribution—this paper proposes the first dynamic Gaussian rasterization-based regularization framework. It innovatively introduces 3D Gaussian splatting into occupancy training: leveraging its differentiable 2D projection for pixel-level semantic supervision, and constructing self-supervised scene flow targets via inter-frame Gaussian parameter changes—eliminating the need for explicit flow annotations. The method comprises sparse semantic Gaussian decoding and dynamic rasterization regularization, significantly improving joint prediction accuracy for both semantic occupancy and scene flow, achieving state-of-the-art results on benchmarks including nuScenes. Moreover, it is fully compatible with mainstream occupancy models and incurs zero inference overhead.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search
• Intelligent Robots: State Estimation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advancements in camera-based occupancy prediction have focused on the simultaneous prediction of 3D semantics and scene flow, a task that presents significant challenges due to specific difficulties, e.g., occlusions and unbalanced dynamic environments. In this paper, we analyze these challenges and their underlying causes. To address them, we propose a novel regularization framework called VoxelSplat. This framework leverages recent developments in 3D Gaussian Splatting to enhance model performance in two key ways: (i) Enhanced Semantics Supervision through 2D Projection: During training, our method decodes sparse semantic 3D Gaussians from 3D representations and projects them onto the 2D camera view. This provides additional supervision signals in the camera-visible space, allowing 2D labels to improve the learning of 3D semantics. (ii) Scene Flow Learning: Our framework uses the predicted scene flow to model the motion of Gaussians, and is thus able to learn the scene flow of moving objects in a self-supervised manner using the labels of adjacent frames. Our method can be seamlessly integrated into various existing occupancy models, enhancing performance without increasing inference time. Extensive experiments on benchmark datasets demonstrate the effectiveness of VoxelSplat in improving the accuracy of both semantic occupancy and scene flow estimation. The project page and codes are available at https://zzy816.github.io/VoxelSplat-Demo/.

Problem

• Improving 3D semantics and scene flow prediction accuracy
• Addressing occlusions and unbalanced dynamic environments
• Enhancing model performance without increasing inference time

Innovation

• Uses 3D Gaussian Splatting for enhanced semantics
• Projects 3D Gaussians to 2D for supervision
• Self-supervised scene flow learning with Gaussians