Unified Sensor Simulation for Autonomous Driving

2026-02-05

AI Summary

Existing sensor simulation methods struggle to jointly model the geometric and appearance characteristics of complex dynamic scenes in autonomous driving, particularly failing to handle the cyclic projection and temporal discontinuities at azimuthal boundaries inherent to spherical sensors such as LiDAR. To address these limitations, this work proposes XSIM, a novel framework built upon an extended 3DGUT splatting approach. XSIM introduces generalized rolling shutter modeling and phase-aware mechanisms to effectively resolve azimuthal boundary discontinuities, and incorporates a dual-opacity 3D Gaussian representation to better align geometry and color distributions. Evaluated on multiple benchmarks—including Waymo Open Dataset, Argoverse 2, and PandaSet—the proposed method significantly outperforms existing approaches, achieving more consistent and photorealistic multi-sensor simulation.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

In this work, we introduce \textbf{XSIM}, a sensor simulation framework for autonomous driving. XSIM extends 3DGUT splatting with a generalized rolling-shutter modeling tailored for autonomous driving applications. Our framework provides a unified and flexible formulation for appearance and geometric sensor modeling, enabling rendering of complex sensor distortions in dynamic environments. We identify spherical cameras, such as LiDARs, as a critical edge case for existing 3DGUT splatting due to cyclic projection and time discontinuities at azimuth boundaries leading to incorrect particle projection. To address this issue, we propose a phase modeling mechanism that explicitly accounts temporal and shape discontinuities of Gaussians projected by the Unscented Transform at azimuth borders. In addition, we introduce an extended 3D Gaussian representation that incorporates two distinct opacity parameters to resolve mismatches between geometry and color distributions. As a result, our framework provides enhanced scene representations with improved geometric consistency and photorealistic appearance. We evaluate our framework extensively on multiple autonomous driving datasets, including Waymo Open Dataset, Argoverse 2, and PandaSet. Our framework consistently outperforms strong recent baselines and achieves state-of-the-art performance across all datasets. The source code is publicly available at \href{https://github.com/whesense/XSIM}{https://github.com/whesense/XSIM}.

Problem

• sensor simulation
• autonomous driving
• 3D Gaussian splatting
• spherical cameras
• azimuth discontinuities

Innovation

• sensor simulation
• 3D Gaussian splatting
• rolling shutter modeling
• spherical camera
• opacity decomposition