MoPD: Mixture-of-Prompts Distillation for Vision-Language Models

2024-12-26

AI Summary

Existing vision-language models exhibit weak generalization to unseen classes in few-/zero-shot settings, primarily due to soft prompts’ over-reliance on training-set priors. To address this, we propose a hybrid prompt distillation framework that (1) introduces a learnable gating network to dynamically select the optimal handcrafted hard prompt—enabling robust knowledge transfer to soft prompts; and (2) integrates contrastive prompt alignment with multi-teacher knowledge distillation to enhance cross-domain consistency. Evaluated on multiple zero-shot transfer benchmarks, our method significantly outperforms state-of-the-art approaches, achieving an average +5.2% accuracy gain on unseen classes while preserving performance on seen classes. Our core contributions are: (i) a learnable hard-prompt gating mechanism; (ii) a synergistic soft/hard prompt distillation paradigm; and (iii) a prompt alignment strategy explicitly designed for unseen-class generalization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Soft prompt learning methods are effective for adapting vision-language models (VLMs) to downstream tasks. Nevertheless, empirical evidence reveals a tendency of existing methods that they overfit seen classes and exhibit degraded performance on unseen classes. This limitation is due to the inherent bias in the training data towards the seen classes. To address this issue, we propose a novel soft prompt learning method, named Mixture-of-Prompts Distillation (MoPD), which can effectively transfer useful knowledge from hard prompts manually hand-crafted (a.k.a. teacher prompts) to the learnable soft prompt (a.k.a. student prompt), thereby enhancing the generalization ability of soft prompts on unseen classes. Moreover, the proposed MoPD method utilizes a gating network that learns to select hard prompts used for prompt distillation. Extensive experiments demonstrate that the proposed MoPD method outperforms state-of-the-art baselines especially on on unseen classes.

Problem

• Visual Language Models
• Generalization
• Novel Object Recognition

Innovation

• MoPD
• Prompt Distillation
• Visual Language Model