Robustifying Vision-Language Models via Test-Time Prompt Adaptation

2026-07-10

AI Summary

This work addresses the significant degradation in robustness of pretrained vision-language models under adversarial perturbations. Existing test-time adaptation methods, which rely on sample-level confidence scores, struggle to distinguish between adversarial misclassifications and genuine semantic variations. To overcome this limitation, the authors propose RITA, a novel framework that elevates prompt adaptation from the sample level to the distribution level. RITA leverages optimal transport to align the distributions of enhanced visual features and textual prototypes, and incorporates a dynamic caching mechanism to refine semantic alignment online. This approach effectively identifies and suppresses adversarial outliers, substantially improving adversarial robustness while preserving high accuracy on clean samples.

Abstract

Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.

Problem

• Vision-Language Models
• Adversarial Robustness
• Test-Time Adaptation
• Distributional Structure
• Semantic Consistency

Innovation

• test-time adaptation
• distribution-level alignment
• optimal transport
• adversarial robustness
• vision-language models