Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

2026-04-21

AI Summary

Existing multimodal test-time adaptation methods lack explicit modeling of class-conditional distributions, and asymmetric distribution shifts across modalities degrade both prediction accuracy and decision boundary reliability. This work addresses these limitations by introducing, for the first time, an explicit class-conditional Gaussian model and proposing an adaptive contrastive asymmetry correction mechanism to effectively mitigate the adverse effects of modality distribution shifts. The proposed approach achieves state-of-the-art performance across multiple multimodal benchmarks under diverse distribution shift scenarios, significantly enhancing model calibration, robustness, and predictive reliability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.

Problem

• multi-modal test-time adaptation
• distribution shift
• category-conditional distribution
• modality asymmetry
• Gaussian discriminant analysis

Innovation

• multi-modal test-time adaptation
• category-conditional distribution
• probabilistic Gaussian model
• modality asymmetry
• adaptive contrastive rectification