AWF: Adaptive Weight Fusion for Enhanced Class Incremental Semantic Segmentation

2024-09-13

arXiv.org

AI Summary

To address severe catastrophic forgetting and the difficulty of balancing old and new knowledge in class-incremental semantic segmentation (CISS), this paper proposes an adaptive weight fusion mechanism. Unlike endpoint weight fusion (EWF), which employs a hand-crafted, fixed fusion coefficient α, our approach introduces a learnable, task-aware α optimized via an alternating training strategy. Within the EWF framework, we jointly incorporate knowledge distillation and incremental fine-tuning to enhance both retention of previously learned classes and robust integration of novel class information. Extensive experiments on mainstream CISS benchmarks demonstrate that our method significantly outperforms EWF and various regularization-based approaches, achieving substantial gains in joint old-and-new class segmentation accuracy. The source code is publicly available.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Class Incremental Semantic Segmentation (CISS) aims to mitigate catastrophic forgetting by maintaining a balance between previously learned and newly introduced knowledge. Existing methods, primarily based on regularization techniques like knowledge distillation, help preserve old knowledge but often face challenges in effectively integrating new knowledge, resulting in limited overall improvement. Endpoints Weight Fusion (EWF) method, while simple, effectively addresses some of these limitations by dynamically fusing the model weights from previous steps with those from the current step, using a fusion parameter alpha determined by the relative number of previously known classes and newly introduced classes. However, the simplicity of the alpha calculation may limit its ability to fully capture the complexities of different task scenarios, potentially leading to suboptimal fusion outcomes. In this paper, we propose an enhanced approach called Adaptive Weight Fusion (AWF), which introduces an alternating training strategy for the fusion parameter, allowing for more flexible and adaptive weight integration. AWF achieves superior performance by better balancing the retention of old knowledge with the learning of new classes, significantly improving results on benchmark CISS tasks compared to the original EWF. And our experiment code will be released on Github.

Problem

• Mitigate catastrophic forgetting in class incremental semantic segmentation
• Improve integration of new knowledge while preserving old knowledge
• Enhance weight fusion adaptability for better CISS performance

Innovation

• Dynamic weight fusion with adaptive parameter
• Alternating training strategy for flexibility
• Improved balance between old and new knowledge