What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

2026-08-25

AI Summary

研究通过PromptSpLiCE方法分析了视觉-语言模型中提示学习前后的自然语言概念变化，解释了提示学习如何改变模型，并与CoOp方法在11个图像分类数据集上进行了评估。

Abstract

Prompt learning adapts vision-language models such as CLIP by optimizing continuous prompt vectors, but the learned prompts are difficult to interpret in natural language. We present PromptSpLiCE, a post-hoc method that expresses each class-conditioned text embedding as a sparse combination of concepts from a fixed natural-language dictionary. Using the same dictionary before and after prompt learning allows us to compare changes in their concept profiles. We evaluate PromptSpLiCE on CoOp, a representative prompt-learning method, across 11 image-classification datasets. The concept profiles change substantially: on average, only 1.6 of the initial top-10 concepts remain in the top 10 after learning. Across datasets, profile change is positively associated with accuracy gain. We also derive a local gradient expression that provides geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.

Problem

• prompt learning
• vision-language models
• natural language interpretation
• concept analysis

Innovation

• Prompt Learning
• Concept Analysis
• Vision-Language Models
• Sparse Combination
• Natural-Language Dictionary