🤖 AI Summary
Addressing open-world continual semantic scene understanding, this work tackles four key challenges: catastrophic forgetting in novel-class incremental learning, background shift, poor class fairness, and weak discriminability for unknown classes. To this end, we propose the first fairness-aware contrastive clustering loss to explicitly mitigate long-tailed class bias. We further introduce an attention-based visual grammar modeling framework—the first to jointly optimize model stability, class fairness, and unknown-class separability. Our method integrates contrastive learning, attention mechanisms, and a continual semantic segmentation architecture. Evaluated on ADE20K, Cityscapes, and Pascal VOC, it achieves state-of-the-art performance: average mIoU gains of +4.2% on long-tailed classes, +18.7% improvement in fairness metrics (e.g., inter-class IoU variance reduction), and significantly enhanced feature discriminability for unknown classes.
📝 Abstract
Continual Learning in semantic scene segmentation aims to continually learn new unseen classes in dynamic environments while maintaining previously learned knowledge. Prior studies focused on modeling the catastrophic forgetting and background shift challenges in continual learning. However, fairness, another major challenge that causes unfair predictions leading to low performance among major and minor classes, still needs to be well addressed. In addition, prior methods have yet to model the unknown classes well, thus resulting in producing non-discriminative features among unknown classes. This work presents a novel Fairness Learning via Contrastive Attention Approach to continual learning in semantic scene understanding. In particular, we first introduce a new Fairness Contrastive Clustering loss to address the problems of catastrophic forgetting and fairness. Then, we propose an attention-based visual grammar approach to effectively model the background shift problem and unknown classes, producing better feature representations for different unknown classes. Through our experiments, our proposed approach achieves State-of-the-Art (SoTA) performance on different continual learning benchmarks, i.e., ADE20K, Cityscapes, and Pascal VOC. It promotes the fairness of the continual semantic segmentation model.
