Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting

2026-04-20

AI Summary

This work addresses the limited adaptability of vision-language models in domain- and category-incremental learning, which stems from using uniform prefix weights across tasks. To overcome this, the authors propose a Dynamic Prefix Weighting (DPW) framework that introduces, for the first time, a gating mechanism based on token importance to dynamically assign prefix weights according to input content. The adapter output is formulated as a residual to the prefix-tuning weights and is activated only when necessary, enabling more fine-grained task adaptation. By integrating parameter-efficient prefix tuning with lightweight adapters, DPW significantly enhances continual learning performance while maintaining low computational overhead, achieving state-of-the-art results on established benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.

Problem

• continual learning
• vision-language models
• prefix-tuning
• domain-class incremental learning
• parameter-efficient adaptation

Innovation

• Dynamic Prefix Weighting
• Vision-Language Models
• Continual Learning
• Prefix Tuning
• Adapters