Completely Weakly Supervised Class-Incremental Learning for Semantic Segmentation

2025-05-16

AI Summary

This paper addresses the problem of fully weakly supervised class-incremental learning for semantic segmentation, where only image-level labels are available. Methodologically, it proposes (1) an uncertainty-aware localizer coupled with multi-foundational-model ensembling to generate robust pixel-level pseudo-labels, and (2) an exemplar-guided data augmentation strategy to mitigate catastrophic forgetting. Its primary contribution is the first framework enabling class-incremental semantic segmentation without any pixel-level annotations—thereby reconciling labeling efficiency with continual learning capability. Experiments demonstrate substantial improvements over existing weakly supervised approaches under both 15-5 and 10-10 PASCAL VOC incremental settings. Moreover, the method achieves competitive accuracy in cross-domain transfer from COCO to VOC, validating its generalizability and effectiveness in realistic weakly supervised continual learning scenarios.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

This work addresses the task of completely weakly supervised class-incremental learning for semantic segmentation to learn segmentation for both base and additional novel classes using only image-level labels. While class-incremental semantic segmentation (CISS) is crucial for handling diverse and newly emerging objects in the real world, traditional CISS methods require expensive pixel-level annotations for training. To overcome this limitation, partially weakly-supervised approaches have recently been proposed. However, to the best of our knowledge, this is the first work to introduce a completely weakly-supervised method for CISS. To achieve this, we propose to generate robust pseudo-labels by combining pseudo-labels from a localizer and a sequence of foundation models based on their uncertainty. Moreover, to mitigate catastrophic forgetting, we introduce an exemplar-guided data augmentation method that generates diverse images containing both previous and novel classes with guidance. Finally, we conduct experiments in three common experimental settings: 15-5 VOC, 10-10 VOC, and COCO-to-VOC, and in two scenarios: disjoint and overlap. The experimental results demonstrate that our completely weakly supervised method outperforms even partially weakly supervised methods in the 15-5 VOC and 10-10 VOC settings while achieving competitive accuracy in the COCO-to-VOC setting.

Problem

• Develops weakly supervised class-incremental semantic segmentation using image-level labels
• Generates robust pseudo-labels via localizer and foundation models with uncertainty
• Mitigates catastrophic forgetting via exemplar-guided data augmentation for diverse images

Innovation

• Generates robust pseudo-labels using localizer and foundation models
• Introduces exemplar-guided data augmentation to prevent forgetting
• Uses only image-level labels for weakly supervised learning