IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting

2025-03-26

AI Summary

To address both forward and backward forgetting in vision-language models under multi-domain class-incremental learning, this paper proposes the Instance-Aware Prompting (IAP) framework. IAP introduces two novel components: Inter-layer Adaptive Gating Prompts (IA-GP) and Class-Distribution-Driven Prompts (IA-CDDP), enabling fine-grained, instance-level adaptive prompt generation. The method integrates parameter-efficient fine-tuning (PEFT), gating mechanisms, instance-level prompt allocation, and task-label confidence modeling—specifically tailored for multimodal Transformer architectures. Extensive experiments across 11 benchmark datasets and three evaluation metrics demonstrate that IAP significantly outperforms existing approaches, effectively mitigating catastrophic forgetting while enhancing adaptation to new tasks. The source code is publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent pre-trained vision-language models (PT-VLMs) often face a Multi-Domain Class-Incremental Learning (MCIL) scenario in practice, where several classes and domains of multi-modal tasks are incrementally arrived. Without access to previously learned tasks and unseen tasks, memory-constrained MCIL suffers from forward and backward forgetting. To alleviate the above challenges, parameter-efficient fine-tuning techniques (PEFT), such as prompt tuning, are employed to adapt the PT-VLM to the diverse incrementally learned tasks. To achieve effective new task adaptation, existing methods only consider the effect of PEFT strategy selection, but neglect the influence of PEFT parameter setting (e.g., prompting). In this paper, we tackle the challenge of optimizing prompt designs for diverse tasks in MCIL and propose an Instance-Aware Prompting (IAP) framework. Specifically, our Instance-Aware Gated Prompting (IA-GP) module enhances adaptation to new tasks while mitigating forgetting by dynamically assigning prompts across transformer layers at the instance level. Our Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) improves the task adaptation process by determining an accurate task-label-related confidence score for each instance. Experimental evaluations across 11 datasets, using three performance metrics, demonstrate the effectiveness of our proposed method. Code can be found at https://github.com/FerdinandZJU/IAP.

Problem

• Optimizing prompt designs for multi-domain class-incremental learning
• Mitigating forgetting in vision-language models via instance-aware prompting
• Enhancing adaptation to new tasks while reducing backward forgetting

Innovation

• Instance-Aware Gated Prompting for dynamic adaptation
• Class-Distribution-Driven Prompting for accurate scoring
• Parameter-efficient fine-tuning for multi-task learning