TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

2026-05-17

AI Summary

This work addresses the vulnerability of large-scale vision-language models, such as CLIP, to imperceptible adversarial perturbations, which poses a critical threat to their safe deployment in open-world settings. To enhance robustness without requiring downstream retraining, the authors propose TAME—a test-time adaptive prompt tuning method that dynamically fuses multiple expert prompts via an input-conditioned mixture-of-experts (MoE) mechanism. TAME integrates input-dependent routing, multi-view prediction entropy minimization, visual token distribution alignment, and MoE regularization. Evaluated across 11 benchmarks, TAME substantially outperforms existing approaches, achieving an average improvement of over 49.1% in adversarial accuracy under AutoAttack while preserving strong generalization on clean samples.

Abstract

Large-scale pre-trained Vision-Language models (VLMs), such as CLIP, exhibit strong zero-shot generalization, yet remain highly vulnerable to imperceptible adversarial perturbations, raising serious safety concerns for open-world deployment. To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense. Building upon our prior Test-Time Adversarial Prompt Tuning (TAPT), TAME introduces an architectural reformulation by replacing TAPT's single adaptive prompt with an input-conditioned Mixture-of-Experts (MoE) framework, enabling more expressive and adaptive defense. Specifically, TAME maintains a bank of learnable expert prompts and employs an input-dependent routing mechanism to aggregate a customized prompt mixture for each unlabeled test sample at inference time. This test-time defense mechanism is driven by three unsupervised objectives: (1) multi-view prediction entropy minimization, (2) layer-wise alignment of visual token statistics to precomputed clean and adversarial reference distributions, and (3) MoE regularization for balanced expert utilization and prompt diversity. We evaluated TAME on 11 benchmark datasets, including ImageNet and 10 additional zero-shot datasets. The results show that TAME improves the zero-shot adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples. TAME also consistently outperforms existing adversarial prompt tuning methods across multiple prompt designs, yielding an average robustness gain of at least 30.2%.

Problem

• Vision-Language Models
• Adversarial Robustness
• Test-Time Defense
• Zero-Shot Generalization
• Adversarial Perturbations

Innovation

• Mixture-of-Experts
• Test-Time Defense
• Adversarial Robustness
• Prompt Tuning
• Vision-Language Models