RealDrive: Retrieval-Augmented Driving with Diffusion Models

2025-05-30

AI Summary

Data-driven driving planners suffer from poor generalization to rare hazardous scenarios and limited trajectory controllability. To address these issues, this paper proposes Retrieval-Augmented Diffusion Planning (RADP), a novel planning framework that integrates retrieval-augmented generation (RAG) with diffusion models. RADP employs a planning-task-driven contrastive learning retriever to precisely select relevant expert trajectories; during the denoising process of a conditional diffusion model, it fuses real-time observations with retrieved demonstrations to enable safe, controllable, and diverse end-to-end trajectory generation. This work is the first to synergize RAG mechanisms with diffusion models for autonomous driving planning, enabling fine-grained behavioral modulation grounded in scenario priors. Evaluated on the Waymo Open Motion dataset, RADP reduces collision rate by 40%, significantly enhancing robustness to long-tail events and improving trajectory diversity.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Learning-based planners generate natural human-like driving behaviors by learning to reason about nuanced interactions from data, overcoming the rigid behaviors that arise from rule-based planners. Nonetheless, data-driven approaches often struggle with rare, safety-critical scenarios and offer limited controllability over the generated trajectories. To address these challenges, we propose RealDrive, a Retrieval-Augmented Generation (RAG) framework that initializes a diffusion-based planning policy by retrieving the most relevant expert demonstrations from the training dataset. By interpolating between current observations and retrieved examples through a denoising process, our approach enables fine-grained control and safe behavior across diverse scenarios, leveraging the strong prior provided by the retrieved scenario. Another key insight we produce is that a task-relevant retrieval model trained with planning-based objectives results in superior planning performance in our framework compared to a task-agnostic retriever. Experimental results demonstrate improved generalization to long-tail events and enhanced trajectory diversity compared to standard learning-based planners -- we observe a 40% reduction in collision rate on the Waymo Open Motion dataset with RAG.

Problem

• Overcoming rigid behaviors in rule-based driving planners
• Addressing rare safety-critical scenarios in data-driven approaches
• Enhancing controllability and diversity in generated trajectories

Innovation

• Retrieval-Augmented Generation for driving
• Diffusion-based planning with expert data
• Task-relevant retrieval model training