Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL

2025-11-16

AI Summary

End-to-end autonomous driving suffers from poor policy transfer across weather domains (e.g., clear → rainy/foggy conditions); existing domain adaptation (DA) approaches rely on target-domain data collection or model retraining, limiting scalability. While prompt-based DA has emerged for perception tasks, it remains inapplicable to closed-loop driving decision-making and requires expert demonstration trajectories. This work pioneers prompt-driven DA for end-to-end driving policies, proposing a test-time adaptation method that requires no model updates—only a small set of generic trajectory demonstrations as contextual prompts. Our core innovation integrates vision-language models with context-aware reinforcement learning to enable dynamic policy adjustment within the CARLA simulator, eliminating dependence on expert annotations or online fine-tuning. Experiments demonstrate substantial improvements in driving safety, efficiency, and ride comfort under adverse weather, consistently outperforming state-of-the-art prompt-based DA baselines.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Application Domains: Mobility, Driving & Flight
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite significant progress and advances in autonomous driving, many end-to-end systems still struggle with domain adaptation (DA), such as transferring a policy trained under clear weather to adverse weather conditions. Typical DA strategies in the literature include collecting additional data in the target domain or re-training the model, or both. Both these strategies quickly become impractical as we increase scale and complexity of driving. These limitations have encouraged investigation into few-shot and zero-shot prompt-driven DA at inference time involving LLMs and VLMs. These methods work by adding a few state-action trajectories during inference to the prompt (similar to in-context learning). However, there are two limitations of such an approach: $(i)$ prompt-driven DA methods are currently restricted to perception tasks such as detection and segmentation and $(ii)$ they require expert few-shot data. In this work, we present a new approach to inference-time few-shot prompt-driven DA for closed-loop autonomous driving in adverse weather condition using in-context reinforcement learning (ICRL). Similar to other prompt-driven DA methods, our approach does not require any updates to the model parameters nor does it require additional data collection in adversarial weather regime. Furthermore, our approach advances the state-of-the-art in prompt-driven DA by extending to closed driving using general trajectories observed during inference. Our experiments using the CARLA simulator show that ICRL results in safer, more efficient, and more comfortable driving policies in the target domain compared to state-of-the-art prompt-driven DA baselines.

Problem

• Addresses domain adaptation challenges in autonomous driving under adverse weather
• Extends prompt-driven methods to closed-loop driving using in-context reinforcement learning
• Eliminates need for model retraining or expert data collection in target domains

Innovation

• In-context reinforcement learning for domain adaptation
• Closed-loop autonomous driving in adverse weather
• No model updates or additional data required