Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models

2026-05-25

AI Summary

Existing vision-language models are prone to losing cross-modal semantic alignment under adversarial perturbations, and prevailing defense strategies—often unidirectional or static—struggle to provide instance-adaptive robustness. This work proposes a fine-tuning-free, closed-loop bidirectional prompting mechanism that, built upon frozen encoders, leverages semantic anchors to enable dynamic feedback between visual and textual modalities for mutual denoising and adaptive prompt refinement. By introducing, for the first time, closed-loop bidirectional interaction guided by semantic anchors, the method fully exploits cross-modal complementarity. It achieves significant improvements in robustness and generalization from base to novel classes across eleven datasets, while maintaining computational efficiency and high accuracy.

Abstract

Vision Language Models adapt well to downstream tasks but are highly vulnerable to adversarial perturbations that disrupt cross-modal semantic alignment. Existing defenses are largely unidirectional or structural, failing to exploit bidirectional cross-modal complementarity and instance-wise adaptive protection. To overcome the limitations of unidirectional and static defenses in adversarial settings, we propose Closed-Loop Bidirectional Prompting, casting robust adaptation as cross-modal agreement recovery via a dynamic feedback loop on frozen encoders. A Semantic Anchor is introduced as a stable prior to constrain cyclic updates and mitigate perturbation-induced feature corruption. Through anchor-based bootstrapping, textual semantics denoise visual representations, while the refined visuals enable instance-adaptive prompt updating, yielding a rectified and robust consensus. Extensive evaluations across 11 datasets validate state-of-the-art robustness and strong base-to-new generalization, while maintaining a favorable trade-off between computational cost and accuracy.

Problem

• adversarial robustness
• vision language models
• cross-modal alignment
• adversarial perturbations
• bidirectional prompting

Innovation

• Closed-Loop Bidirectional Prompting
• Adversarial Robustness
• Vision Language Models
• Semantic Anchor
• Cross-modal Alignment