ProTPS: Prototype-Guided Text Prompt Selection for Continual Learning

2026-04-01

AI Summary

This work addresses the challenges in continual learning where conventional text prompting methods struggle to capture the distinctive semantics of new categories and are prone to catastrophic forgetting. To mitigate these issues, we propose a prototype-guided textual prompt learning mechanism that leverages class-specific visual prototypes to dynamically steer the optimization of learnable textual prompts, thereby disentangling semantic features and enhancing the model’s discriminative capacity for novel classes. We introduce Marine112, a real-world long-tailed marine species dataset, and evaluate our approach under three continual learning settings: class-incremental, cross-dataset, and class-domain incremental. Experimental results demonstrate that our method significantly outperforms state-of-the-art approaches, achieving performance close to the theoretical upper bound. The code and the Marine112 dataset are publicly released.

Abstract

For continual learning, text-prompt-based methods leverage text encoders and learnable prompts to encode semantic features for sequentially arrived classes over time. A common challenge encountered by existing works is how to learn unique text prompts, which implicitly carry semantic information of new classes, so that the semantic features of newly arrived classes do not overlap with those of trained classes, thereby mitigating the catastrophic forgetting problem. To address this challenge, we propose a novel approach Prototype-guided Text Prompt Selection (ProTPS)'' to intentionally increase the training flexibility thus encouraging the learning of unique text prompts. Specifically, our ProTPS learns class-specific vision prototypes and text prompts. Vision prototypes guide the selection and learning of text prompts for each class. We first evaluate our ProTPS in both class incremental (CI) setting and cross-datasets continual (CDC) learning setting. Because our ProTPS achieves performance close to the upper bounds, we further collect a real-world dataset with 112 marine species collected over a span of six years, named Marine112, to bring new challenges to the community. Marine112 is authentically suited for the class and domain incremental (CDI) learning setting and is under natural long-tail distribution. The results under three settings show that our ProTPS performs favorably against the recent state-of-the-art methods. The implementation code and Marine112 dataset will be released upon the acceptance of our paper.

Problem

• continual learning
• text prompt
• catastrophic forgetting
• semantic features
• class incremental learning

Innovation

• Prototype-guided Prompt Learning
• Continual Learning
• Text Prompt Selection
• Vision Prototype
• Catastrophic Forgetting