Analytic Continual Test-Time Adaptation for Multi-Modality Corruption

2024-10-29

arXiv.org

AI Summary

To address catastrophic forgetting, reliability bias, and insufficient robustness under multimodal corruptions in multimodal continual test-time adaptation (MM-CTTA), this paper proposes the first TTA framework incorporating analytic learning. Methodologically: (1) Analytic classifiers (ACs) are constructed to enable parameter-interpretable, gradient-free class discrimination, fundamentally mitigating forgetting; (2) A dynamic selection mechanism (DSM) jointly optimizes cross-modal reliable sample identification and weighted fusion via soft pseudo-labeling (SPS); (3) A multimodal dynamic analytic adapter (MDAA) achieves modality-adaptive alignment. Evaluated under label-free, continual domain shifts—e.g., weather variations and sensor failures—the method significantly curbs error accumulation, achieving state-of-the-art performance. It establishes a novel paradigm for MM-CTTA that is interpretable, robust, and sustainable.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Test-Time Adaptation (TTA) aims to help pre-trained model bridge the gap between source and target datasets using only the pre-trained model and unlabelled test data. A key objective of TTA is to address domain shifts in test data caused by corruption, such as weather changes, noise, or sensor malfunctions. Multi-Modal Continual Test-Time Adaptation (MM-CTTA), an extension of TTA with better real-world applications, further allows pre-trained models to handle multi-modal inputs and adapt to continuously-changing target domains. MM-CTTA typically faces challenges including error accumulation, catastrophic forgetting, and reliability bias, with few existing approaches effectively addressing these issues in multi-modal corruption scenarios. In this paper, we propose a novel approach, Multi-modality Dynamic Analytic Adapter (MDAA), for MM-CTTA tasks. We innovatively introduce analytic learning into TTA, using the Analytic Classifiers (ACs) to prevent model forgetting. Additionally, we develop Dynamic Selection Mechanism (DSM) and Soft Pseudo-label Strategy (SPS), which enable MDAA to dynamically filter reliable samples and integrate information from different modalities. Extensive experiments demonstrate that MDAA achieves state-of-the-art performance on MM-CTTA tasks while ensuring reliable model adaptation.

Problem

• Address domain shifts from multi-modality corruption during test time
• Mitigate catastrophic forgetting in continual test-time adaptation
• Dynamically integrate reliable information from multiple modalities

Innovation

• Multi-modality Dynamic Analytic Adapter (MDAA)
• Analytic Classifiers (ACs) mitigate catastrophic forgetting
• Dynamic Late Fusion Mechanism (DLFM) integrates modalities