T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models

2026-06-22

AI Summary

Current vision-language models exhibit insufficient robustness under adversarial perturbations, and existing test-time adaptation methods lack direct correction of corrupted visual representations. This work proposes T-VSS, the first lightweight test-time adaptation approach that operates directly in the visual feature space. By constructing sample-specific low-rank subspaces and integrating reliability-weighted entropy minimization, T-VSS enables efficient and targeted feature correction while avoiding noisy updates across the entire feature space. The method significantly enhances adversarial robustness on ImageNet, fine-grained, and ImageNet-OOD benchmarks, all while preserving high accuracy on clean samples and achieving superior computational efficiency.

Abstract

Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations. Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptations.

Problem

• adversarial robustness
• vision-language models
• test-time adaptation
• visual representation
• adversarial perturbations

Innovation

• test-time adaptation
• visual subspace steering
• adversarial robustness
• vision-language models
• low-rank subspace