NudgeVAD: Language-Nudged End-to-End Driving via FiLM Residuals

2026-05-23

AI Summary

This work addresses the limited utility of natural language instructions in existing end-to-end driving systems when high-level commands are already reliable. The authors propose NudgeVAD, a framework that introduces a language-conditioned FiLM residual module atop a frozen primary planner. By initializing the FiLM layers as identity mappings and the residual heads to zero, the design ensures that language acts solely as a calibrative fine-tuning signal. This approach preserves baseline planning performance while significantly improving robustness under unreliable command conditions: with random commands, language-based fine-tuning reduces the ADE6s to 2.806 meters, outperforming the language-free model by 0.312 meters. In contrast, gains under reliable commands remain marginal, confirming the framework’s “on-demand intervention” principle.

Abstract

Natural-language instructions promise controllable end-to-end driving, but their benefit can be hidden when planners already receive reliable high-level commands. We propose NudgeVAD, a frozen-planner residual framework that uses language as a calibrated nudge to a VAD trajectory. With identity-initialized FiLM and a zero-initialized residual head, NudgeVAD is equivalent to the frozen planner at initialization, so learned deviations arise only from language-conditioned residuals. We evaluate NudgeVAD along a command-reliability axis. With reliable commands, language improves the initial planner but becomes nearly redundant once compared against VAD-FT (UNCOND), a compute-matched VAD model fine-tuned without language. With random commands, however, language becomes essential: detaching text degrades ADE6s to 3.166 m, while NudgeVAD with text recovers 2.806 m and outperforms VAD-FT (UNCOND) by 0.312 m. These results show that language is not universally additive; it is most valuable when the categorical command channel is unreliable.

Problem

• end-to-end driving
• natural-language instructions
• trajectory prediction
• command reliability
• autonomous driving

Innovation

• NudgeVAD
• language-conditioned driving
• FiLM residuals
• end-to-end autonomous driving
• command reliability