A Similarity Paradigm Through Textual Regularization Without Forgetting

2025-02-20

AI Summary

To address catastrophic forgetting of general knowledge and degraded cross-distribution generalization in vision-language models (VLMs) caused by overfitting to textual prompts during prompt learning, this paper proposes the **forgetting-free similarity paradigm SPTR**. Methodologically, SPTR introduces the first optimal-transport-based text regularization mechanism, which constrains fine-tuning from deviating from the semantic structure of predefined hand-crafted prompts; it further designs a dual-path similarity modeling framework—comprising natural alignment and adversarial alignment—to continuously activate shared knowledge across multiple hand-crafted prompts. Evaluated on 11 benchmarks across four generalization settings—few-shot learning, base-to-novel class transfer, cross-dataset generalization, and domain generalization—SPTR consistently outperforms existing prompt learning methods. It is the first approach to systematically mitigate knowledge forgetting in prompt tuning, achieving significant improvements in zero-shot transfer robustness.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Prompt learning has emerged as a promising method for adapting pre-trained visual-language models (VLMs) to a range of downstream tasks. While optimizing the context can be effective for improving performance on specific tasks, it can often lead to poor generalization performance on unseen classes or datasets sampled from different distributions. It may be attributed to the fact that textual prompts tend to overfit downstream data distributions, leading to the forgetting of generalized knowledge derived from hand-crafted prompts. In this paper, we propose a novel method called Similarity Paradigm with Textual Regularization (SPTR) for prompt learning without forgetting. SPTR is a two-pronged design based on hand-crafted prompts that is an inseparable framework. 1) To avoid forgetting general textual knowledge, we introduce the optimal transport as a textual regularization to finely ensure approximation with hand-crafted features and tuning textual features. 2) In order to continuously unleash the general ability of multiple hand-crafted prompts, we propose a similarity paradigm for natural alignment score and adversarial alignment score to improve model robustness for generalization. Both modules share a common objective in addressing generalization issues, aiming to maximize the generalization capability derived from multiple hand-crafted prompts. Four representative tasks (i.e., non-generalization few-shot learning, base-to-novel generalization, cross-dataset generalization, domain generalization) across 11 datasets demonstrate that SPTR outperforms existing prompt learning methods.

Problem

• Improves generalization in prompt learning
• Prevents forgetting hand-crafted prompt knowledge
• Enhances model robustness across diverse tasks

Innovation

• Optimal transport textual regularization
• Similarity paradigm alignment scores
• Hand-crafted prompts generalization enhancement