Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

2026-04-30

AI Summary

This study systematically investigates the cross-architecture transferability and safety risks of physical adversarial attacks against vision-language models (VLMs) in autonomous driving. Focusing on real-world road scenarios, the authors design physically realizable adversarial patches and, through a cross-model transferability matrix combined with temporal analysis of critical decision windows, reveal for the first time that VLMs exhibit high adversarial transferability. Experiments demonstrate that non-targeted optimized attacks achieve average transfer rates of 0.815 and 0.833 in crosswalk and highway scenarios, respectively, with cross-model attack success rates ranging from 73% to 91%. Moreover, these attacks persistently disrupt driving decisions in 64.7%–79.4% of critical frames, underscoring their substantial real-world safety implications.

Abstract

Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across different VLM architectures, is not well understood and poses a practical risk when attackers do not know which model a vehicle uses. We address this gap with a systematic cross-architecture study of adversarial transferability in VLM-based driving, evaluating three representative architectures (Dolphins, OmniDrive, and LeapVAD) using physically realizable patches placed on roadside infrastructure in both crosswalk and highway scenarios. Our transfer-matrix evaluation shows high cross-architecture effectiveness, with transfer rates of 73-91% (mean TR = 0.815 for crosswalk and 0.833 for highway) and sustained frame-level manipulation over 64.7-79.4% of the critical decision window even when patches are not optimized for the target model.

Problem

• adversarial transferability
• vision-language models
• autonomous driving
• physical adversarial attacks
• cross-architecture

Innovation

• adversarial transferability
• vision-language models
• autonomous driving
• physical adversarial attacks
• cross-architecture evaluation