ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation

2026-04-18

AI Summary

Existing approaches struggle to generate diverse and realistic driving scenes controllably through visual or linguistic instructions. This work proposes a vision-language-controlled generative framework in a vectorized latent space that accepts either text or image inputs to synthesize temporally consistent 3D driving scenarios, including road maps, dynamic traffic agents, and ego-vehicle viewpoints. The key innovation lies in enabling fine-grained cross-modal control over vectorized driving scenes for the first time, achieved through a cross-global control mechanism that integrates cross-attention with a lightweight global context branch, allowing precise manipulation of road layouts and traffic states while preserving realism. Experiments demonstrate that the method significantly outperforms existing approaches in both control accuracy and generation fidelity, supports long-horizon scene continuation, and introduces the first publicly available vectorized map dataset annotated with textual descriptions.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

We introduce ScenarioControl, the first vision-language control mechanism for learned driving scenario generation. Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts - including map, 3D boxes of reactive actors over time, pedestrians, driving infrastructure, and ego camera observations. The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly. To connect multimodal control with sparse vectorized scene elements, we propose a cross-global control mechanism that integrates crossattention with a lightweight global-context branch, enabling fine-grained control over road layout and traffic conditions while preserving realism. The method produces temporally consistent scenario rollouts from the perspectives different actors in the scene, supporting long-horizon continuation of driving scenarios. To facilitate training and evaluation, we release a dataset with text annotations aligned to vectorized map structures. Extensive experiments validate that the control adherence and fidelity of ScenarioControl compare favorable to all tested methods across all experiments. Project webpage: https://light.princeton.edu/ScenarioControl

Problem

• vision-language control
• driving scenario generation
• vectorized latent space
• multimodal control
• temporal consistency

Innovation

• vision-language control
• vectorized latent space
• cross-global control
• scenario generation
• autonomous driving simulation