Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

2026-08-03

AI Summary

This work addresses the severe performance degradation of vision-language models under distribution shifts at test time, a challenge exacerbated by existing test-time adaptation methods that often suffer from error amplification and bias accumulation, leading to semantic geometry distortion and mode collapse. To mitigate these issues, the authors propose LMR, a lightweight single-step adaptation framework that integrates a Protected Margin Restoration (PMR) optimization objective, an Adaptive Margin (AM) controller, and a Bias Correction (BC) mechanism. Operating without labels, LMR simultaneously restores local semantic structures and suppresses bias propagation through sample-level margin recovery and a two-stage flow-level stabilization strategy. Extensive experiments demonstrate that LMR significantly outperforms current approaches on benchmarks such as CIFAR-C and ImageNet-C, maintaining robust and efficient performance even under challenging conditions like small batch sizes.

Abstract

Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.

Problem

• test-time adaptation
• distribution shift
• vision-language models
• semantic geometry
• mode collapse

Innovation

• Test-Time Adaptation
• Vision-Language Models
• Local Margin Restoration
• Semantic Geometry
• Mode Collapse