LidarDM: Generative LiDAR Simulation in a Generated World

2024-04-03

IEEE International Conference on Robotics and Automation

AI Summary

Existing methods for generating realistic, layout-aware, physically plausible, and temporally coherent 4D LiDAR video in autonomous driving simulation remain inadequate. Method: This paper introduces the first generative 4D LiDAR world model tailored for driving scenes. It employs an integrated framework that leverages latent diffusion models for 3D scene generation, jointly models dynamic agent motion, and captures spatiotemporal (4D) point cloud sequences; LiDAR video is then synthesized via differentiable sensor rendering. Contribution/Results: Our approach pioneers driving-semantic-guided 4D LiDAR generation, uniquely ensuring layout consistency, physical interpretability, and temporal coherence. Quantitative and qualitative evaluations demonstrate significant improvements over prior art in realism, temporal continuity, and structural fidelity. The generated LiDAR sequences effectively support downstream perception model training and evaluation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models. We release our source code and checkpoints at https://github.com/vzyrianov/LidarDM

Problem

• Generates realistic LiDAR videos for autonomous driving simulation
• Creates 4D LiDAR point clouds with temporal coherence
• Simulates sensory observations in a generative virtual world

Innovation

• Generative LiDAR simulation with driving scenarios
• 4D world generation for coherent point cloud sequences
• Latent diffusion models for 3D scene synthesis