Problem
Research questions and friction points this paper is trying to address.
instruction robustness
autonomous driving
language-conditioned control
counterfactual evaluation
embodied foundation models
ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving
🤖 AI Summary
Current language-driven autonomous driving systems exhibit insufficient robustness when confronted with variations in phrasing, ambiguity, noise, or misleading instructions, and lack systematic evaluation of reliability at the instruction level. This work proposes ICR-Drive, a diagnostic framework that formalizes and quantifies counterfactual robustness of language commands for the first time. By generating four controlled types of instruction variants—rewritten, ambiguous, noisy, and misleading—within the CARLA closed-loop simulation under fixed-route conditions, the framework isolates the impact of linguistic perturbations on driving performance. Experiments reveal that state-of-the-art systems such as LMDrive and BEVDriver suffer significant performance degradation and exhibit distinct failure modes even under minor instruction perturbations, exposing critical reliability gaps in current embodied foundation models for safety-critical scenarios.
📝 Abstract
Recent progress in vision-language-action (VLA) models has enabled language-conditioned driving agents to execute natural-language navigation commands in closed-loop simulation, yet standard evaluations largely assume instructions are precise and well-formed. In deployment, instructions vary in phrasing and specificity, may omit critical qualifiers, and can occasionally include misleading, authority-framed text, leaving instruction-level robustness under-measured. We introduce ICR-Drive, a diagnostic framework for instruction counterfactual robustness in end-to-end language-conditioned autonomous driving. ICR-Drive generates controlled instruction variants spanning four perturbation families: Paraphrase, Ambiguity, Noise, and Misleading, where Misleading variants conflict with the navigation goal and attempt to override intent. We replay identical CARLA routes under matched simulator configurations and seeds to isolate performance changes attributable to instruction language. Robustness is quantified using standard CARLA Leaderboard metrics and per-family performance degradation relative to the baseline instruction. Experiments on LMDrive and BEVDriver show that minor instruction changes can induce substantial performance drops and distinct failure modes, revealing a reliability gap for deploying embodied foundation models in safety-critical driving.
Innovation
Methods, ideas, or system contributions that make the work stand out.
instruction counterfactual robustness
language-conditioned autonomous driving
controlled instruction perturbation
misleading instructions
embodied foundation models
🔎 Similar Papers
No similar papers found.
