On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

2026-07-08

AI Summary

This study investigates the vulnerability mechanisms of vision-language models under adversarial attacks, with a focus on how the spectral structure of intermediate linear transformations influences model robustness. To this end, the authors propose a white-box Spectral Subspace-Guided Attack (SSGRA), which enhances attack efficacy by aligning intermediate representations with the subspace spanned by right singular vectors. This work is the first to reveal the adversarial fragility of vision-language models from the perspective of spectral subspaces, achieving higher attack success rates than existing baselines. Moreover, it offers novel theoretical insights and a principled technical pathway toward understanding and improving the robustness of such models.

Abstract

Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.

Problem

• adversarial vulnerability
• vision-language models
• spectral subspaces
• intermediate representations
• transformer-based models

Innovation

• spectral subspaces
• adversarial vulnerability
• vision-language models
• singular vectors
• white-box attack