FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

2026-06-18

AI Summary

Existing diffusion models struggle to simultaneously ensure multi-view consistency and temporal coherence when generating autonomous driving scenes, and exhibit poor generalization under adverse weather or rare scenarios, often suffering from catastrophic forgetting of pre-trained knowledge. This work proposes a fine-tuning-free controllable generation framework that, while keeping the pre-trained diffusion model frozen, integrates textual prompts with driving stack signals and introduces a knowledge-preserving spatiotemporal attention mechanism along with object-level constraints. This enables globally consistent and high-fidelity generation of multi-view, temporally coherent driving scenes. Evaluated on nuScenes, the generated augmented data significantly enhances downstream autonomous driving model performance, demonstrating notably improved robustness under challenging conditions such as nighttime and rain.

Abstract

Synthetic data for autonomous driving is surging, powered by diffusion models that promise scalable scene generation. Yet key obstacles remain, as enforcing multi-view and temporal consistency often relies on backbone fine-tuning or added layers, which erodes pre-trained knowledge and weakens text alignment. Models also stay close to the training distribution, struggling under adverse weather and unseen configurations, and fidelity favors frequent over rare classes. We address these gaps with FrozenDrive, a controllable generative framework that preserves a pretrained diffusion models knowledge while achieving strong consistency. FrozenDrive conditions on rich driving-stack signals and text prompts, and introduces knowledge-preserving spatio-temporal attention to impose cross-view alignment and temporal coherence in a single pass within a parameter-free frozen diffusion backbone. An additional object-focused constraint improves per-object fidelity for rare categories. Without any weather- or scene-specific fine-tuning, our model synthesizes globally coherent multi-view driving scenes from text, particularly under adverse and rare conditions, and surpasses prior baselines. On nuScenes, FrozenDrive augmented data significantly improves AD models performance, especially at night and in rain, demonstrating stronger robustness when trained with our scenario-targeted data.

Problem

• autonomous driving
• diffusion models
• multi-view consistency
• temporal coherence
• data augmentation

Innovation

• frozen diffusion
• zero-shot generation
• spatio-temporal consistency
• text-guided synthesis
• data augmentation