T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation

2026-06-29

AI Summary

This work addresses the challenges of over-smoothing and limited controllability in text-to-LiDAR scene generation, primarily caused by data scarcity. To this end, the authors propose T2LDM++, a novel framework that incorporates a lightweight, inference-decoupled self-conditioning mechanism to guide the denoising network with reconstruction-oriented soft supervision, thereby enhancing geometric awareness. Additionally, a directional positional prior is introduced to mitigate street-level distortions and enable multimodal conditional generation. The study also presents the first high-quality Text-LiDAR dataset alongside dedicated metrics for evaluating controllability. Experimental results demonstrate that T2LDM++ generates LiDAR scenes with rich details and high realism under both unconditional and conditional settings, significantly outperforming existing methods on the newly established benchmark.

Abstract

Recent progress in Text-to-Image generation benefits from large-scale Text-Image pairs. However, the scarcity of Text-LiDAR pairs often causes over-smoothed scenes and limited controllability. In this paper, we rethink the limitations of Text-LiDAR generation task, focusing on alleviating insufficient training priors and constructing controllable Text-LiDAR data. We propose a \textbf{T}ext-\textbf{to}-\textbf{L}iDAR \textbf{D}iffusion \textbf{M}odel for LiDAR scene generation, T2LDM++, with a Self-Conditioned Representation Guidance (SCRG). Specifically, to alleviate object over-smoothing, SCRG employs a Guidance Network (GN) to provide reconstruction-based soft supervision to the Denoising Network (DN). This enables DN to learn geometry-aware representations through reconstruction guidance, leading to more accurate denoising in DDPMs. Meanwhile, through analysis and design, SCRG exhibits more effective and lightweight, while decoupled in inference, avoiding computational overhead. Furthermore, we construct two high-quality Text-LiDAR benchmarks ($>$100K samples) using a generalized strategy of geometric annotations, along with a controllability metric. Moreover, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, T2LDM++ supports multiple conditions, including (Semantic, Box, BEV, Camera)-to-LiDAR, Sparse-to-Dense, and Dense-to-Sparse generation, by learning a control encoder via frozen DN. With effective prior modeling and high-quality Text-LiDAR benchmarks, T2LDM++ can generate realistic LiDAR scenes with rich geometric details in unconditional and conditional settings.

Problem

• Text-to-LiDAR
• LiDAR scene generation
• data scarcity
• over-smoothing
• controllability

Innovation

• Self-Conditioned Representation Guidance
• Text-to-LiDAR Generation
• Diffusion Model
• Geometric Prior
• Controllable Scene Synthesis