Learning Probabilistic Prompt for Continual Learning

2026-07-06

AI Summary

This work addresses the challenge of prompt collapse in prompt-based continual learning, which hinders the capture of diverse data distributions. To overcome this limitation, the paper proposes modeling prompts as probabilistic distributions and constructs a mixture distribution from which diverse prompts are sampled, thereby enhancing the representation of multimodal image features across task sequences. Additionally, a distributional regularization loss is introduced to effectively suppress abrupt shifts in the prompt distribution during training. The proposed method achieves significant performance gains over existing approaches on standard continual learning benchmarks, including ImageNet-R, CIFAR-100, and CUB-200, demonstrating both its effectiveness and strong generalization capability.

Abstract

Continual learning aims to progressively learn from a sequence of tasks, each containing a disjoint subset of classes, while preserving previously learned knowledge. Prompt-based continual learning methods propose to learn a small set of parameters, i.e., prompts, by associating them with a query feature of an input image. These methods optimize the prompts, attempting to represent diverse patterns of images. However, we have observed that existing prompt-based methods suffer from a prompt collapse problem, that is, the prompts tend to be highly similar to each other, thereby failing to capture the diverse data distributions in continual learning scenarios. To address this issue, we propose in this paper a novel prompt-based continual learning framework that captures diverse patterns of images across a sequence of tasks. To this end, we model each prompt as a probabilistic distribution and construct a mixture of these distributions, from which we sample diverse prompts. This enables our model to effectively capture highly diverse image distributions in the continual learning process. We also present a distribution regularization loss to prevent abrupt changes in the prompt distributions throughout the training process. We show extensive experimental results for continual learning on standard benchmarks, including ImageNet-R, CIFAR-100, and CUB-200, demonstrating the effectiveness of our framework.

Problem

• continual learning
• prompt collapse
• probabilistic prompt
• data distribution diversity

Innovation

• probabilistic prompt
• continual learning
• prompt collapse
• distribution regularization
• mixture of distributions