Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

2025-06-10

AI Summary

To address the scarcity of real-world edge-case driving data and high annotation costs in autonomous driving systems, this paper introduces Cosmos-Drive—the first world foundation model specialized for driving, built upon NVIDIA Cosmos. Our method leverages controllable video synthesis, multi-view consistency modeling, and spatiotemporal joint generation to construct a high-fidelity, multi-view, spatiotemporally consistent synthetic driving data pipeline. This approach significantly mitigates long-tail distribution challenges, yielding improved generalization across 3D lane detection, 3D object detection, and end-to-end driving policy learning. We publicly release the complete toolchain, synthetic dataset, and model weights, establishing a scalable, reproducible synthetic-data paradigm for safety-critical AI systems. (126 words)

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams

Problem

• Generates synthetic driving data for rare edge cases
• Enhances perception and driving policy training
• Mitigates long-tail distribution in AV datasets

Innovation

• Synthetic data generation pipeline for AV
• NVIDIA Cosmos-driven high-fidelity video generation
• Open-source toolkit for diverse driving datasets