Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

2026-06-21

AI Summary

This work addresses the overfitting of base classes and degraded generalization to novel classes in few-shot scenarios when fine-tuning only class-specific prompts in CLIP. To mitigate this, the authors propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization approach that operates under frozen CLIP encoders. CCPL aligns learnable prompts with predefined concept-level textual prototypes through shared context tokens, incorporates concept dropout for regularization, and employs a controllable logit fusion mechanism. Experiments demonstrate that CCPL improves the harmonic mean of base and novel class accuracy by 0.6% on DTD and 2.9% on EuroSAT, while incurring only a marginal 0.1% drop on OxfordPets, thereby validating its effectiveness and delineating its applicability boundaries in enhancing cross-category generalization.

Abstract

Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.

Problem

• few-shot learning
• prompt learning
• CLIP adaptation
• overfitting
• generalization

Innovation

• prompt learning
• concept regularization
• few-shot adaptation
• CLIP
• text-space alignment