Class-aware Domain Knowledge Fusion and Fission for Continual Test-Time Adaptation

2025-10-14

AI Summary

To address catastrophic forgetting of historical knowledge, insufficient learning of novel knowledge, and interference from detrimental historical knowledge in continual test-time adaptation (CTTA), this paper proposes a class-aware domain knowledge fission and fusion mechanism. Without access to downstream training data, our approach dynamically disentangles discriminative knowledge from new domains and selectively integrates it into the historical knowledge pool via a class-aware domain prompt pool, a Knowledge Fission Module (KFI), and a Knowledge Fusion Module (KFU). We further design a greedy dynamic merging strategy that mitigates negative transfer while preserving computational efficiency. Experiments on ImageNet-C demonstrate that our method significantly outperforms existing CTTA approaches, effectively alleviating both forgetting and overfitting. It exhibits superior stability and generalization across multiple sequential domain shifts.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Continual Test-Time Adaptation (CTTA) aims to quickly fine-tune the model during the test phase so that it can adapt to multiple unknown downstream domain distributions without pre-acquiring downstream domain data. To this end, existing advanced CTTA methods mainly reduce the catastrophic forgetting of historical knowledge caused by irregular switching of downstream domain data by restoring the initial model or reusing historical models. However, these methods are usually accompanied by serious insufficient learning of new knowledge and interference from potentially harmful historical knowledge, resulting in severe performance degradation. To this end, we propose a class-aware domain Knowledge Fusion and Fission method for continual test-time adaptation, called KFF, which adaptively expands and merges class-aware domain knowledge in old and new domains according to the test-time data from different domains, where discriminative historical knowledge can be dynamically accumulated. Specifically, considering the huge domain gap within streaming data, a domain Knowledge FIssion (KFI) module is designed to adaptively separate new domain knowledge from a paired class-aware domain prompt pool, alleviating the impact of negative knowledge brought by old domains that are distinct from the current domain. Besides, to avoid the cumulative computation and storage overheads from continuously fissioning new knowledge, a domain Knowledge FUsion (KFU) module is further designed to merge the fissioned new knowledge into the existing knowledge pool with minimal cost, where a greedy knowledge dynamic merging strategy is designed to improve the compatibility of new and old knowledge while keeping the computational efficiency. Extensive experiments on the ImageNet-C dataset verify the effectiveness of our proposed method against other methods.

Problem

• Adapting models to multiple unknown domain distributions during test phase
• Reducing catastrophic forgetting while preventing insufficient new knowledge learning
• Managing computational efficiency and knowledge compatibility across domain shifts

Innovation

• Class-aware domain knowledge fusion and fission method
• Adaptively separates new domain knowledge from old
• Merges new knowledge efficiently into existing pool