CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving

2025-03-09

AI Summary

To address the challenge of high-fidelity 4D rendering for dynamic traffic scenes in closed-loop autonomous driving simulation, this paper proposes a context- and deformation-aware dynamic Gaussian splatting method. Our approach introduces three key contributions: (1) a novel semantic segmentation foundation model–driven 4D Gaussian semantic embedding mechanism; (2) an explicit Gaussian ellipsoid-based temporal deformation modeling and cross-frame fusion encoding framework, enabling fine-grained motion representation while preserving semantic consistency; and (3) a self-supervised 4D semantic feature learning scheme coupled with multi-feature aggregation rendering. Evaluated on dynamic 4D reconstruction and novel-view synthesis tasks, our method significantly outperforms existing self-supervised approaches—particularly in motion-edge sharpness, occlusion handling, and semantic detail recovery. The resulting high-fidelity, semantically coherent 4D reconstructions effectively support high-assurance, end-to-end simulation-based validation of autonomous driving algorithms.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Dynamic scene rendering opens new avenues in autonomous driving by enabling closed-loop simulations with photorealistic data, which is crucial for validating end-to-end algorithms. However, the complex and highly dynamic nature of traffic environments presents significant challenges in accurately rendering these scenes. In this paper, we introduce a novel 4D Gaussian Splatting (4DGS) approach, which incorporates context and temporal deformation awareness to improve dynamic scene rendering. Specifically, we employ a 2D semantic segmentation foundation model to self-supervise the 4D semantic features of Gaussians, ensuring meaningful contextual embedding. Simultaneously, we track the temporal deformation of each Gaussian across adjacent frames. By aggregating and encoding both semantic and temporal deformation features, each Gaussian is equipped with cues for potential deformation compensation within 3D space, facilitating a more precise representation of dynamic scenes. Experimental results show that our method improves 4DGS's ability to capture fine details in dynamic scene rendering for autonomous driving and outperforms other self-supervised methods in 4D reconstruction and novel view synthesis. Furthermore, CoDa-4DGS deforms semantic features with each Gaussian, enabling broader applications.

Problem

• Improves dynamic scene rendering for autonomous driving simulations.
• Incorporates context and temporal deformation awareness in 4D Gaussian Splatting.
• Enhances 4D reconstruction and novel view synthesis accuracy.

Innovation

• 4D Gaussian Splatting with context awareness
• Self-supervised 4D semantic feature embedding
• Temporal deformation tracking for dynamic scenes