Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

2026-06-24

AI Summary

This work addresses the degradation in accuracy and inference efficiency under zero-shot noisy testing scenarios caused by out-of-distribution (OOD) anomalous samples. The authors propose a training-free dual-distribution estimation framework that pioneers a shift from instance-level learning to online distribution modeling. By separately estimating a positive feature Gaussian distribution and a negative label distribution for each class, the method enables efficient OOD filtering and in-distribution (ID) classification. Integrated with contrastive score calibration and a negative label mining strategy, the framework achieves a 3.70% improvement in harmonic mean accuracy and a 6.20% reduction in FPR95 for OOD detection on ImageNet, while maintaining highly efficient and scalable online inference capabilities.

Abstract

While test-time adaptation (TTA) empowers vision-language models to adapt without costly retraining, it remains highly vulnerable to out-of-distribution (OOD) outliers prevalent in real-world applications. This discrepancy motivates Noisy TTA (NTTA), an online task to filter noisy OOD samples on the fly while maximizing in-distribution (ID) classification accuracy. Existing zero-shot NTTA approaches typically rely on test-time discriminative training, leading to overconfident misclassifications and significantly degraded inference efficiency. To address these limitations, we propose a novel framework named Dual Distribution Estimation (DDE), shifting the zero-shot NTTA paradigm from instance-level learning to training-free Gaussian distribution modeling. DDE incorporates two novel modules: Positive Feature Distribution Estimation (PFDE) and Negative Label Distribution Estimation (NLDE). PFDE explicitly models class-wise inclusion and exclusion Gaussian distributions to formulate a calibrated contrastive score, robustly enhancing ID accuracy. In parallel, NLDE improves OOD identification by explicitly modeling the negative label distribution to mine highly discriminative labels, effectively mitigating spurious correlations. Extensive experiments show that on the large-scale ImageNet benchmark, DDE achieves an improvement of 3.70\% in harmonic mean accuracy and reduces the FPR95 for OOD detection by 6.20\%, while ensuring highly scalable and efficient online inference. Furthermore, DDE is zero-shot and training-free, demonstrating remarkable robustness in data-scarce scenarios. Codes are available at https://github.com/ZhuWenjie98/DDE.

Problem

• test-time adaptation
• out-of-distribution detection
• zero-shot learning
• noisy test-time adaptation
• vision-language models

Innovation

• Dual Distribution Estimation
• Zero-shot Noisy Test-Time Adaptation
• Gaussian Distribution Modeling
• Out-of-Distribution Detection
• Training-free Adaptation