When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning

2026-05-10

AI Summary

In few-shot vision-language prompt learning with frozen backbones, adaptive gating mechanisms often fail to outperform fixed prompts due to gradient magnitude imbalance and gating collapse. This work systematically investigates this issue through gradient analysis, controlled comparative experiments, and multi-architecture ablations across multiple datasets. We reveal for the first time that indiscriminately increasing the complexity of adaptive modules can trigger mechanism collapse rather than improvement. The study delineates the boundary conditions under which adaptive gating is effective and demonstrates that such mechanisms frequently produce near-constant outputs with vanishingly small gradients, yielding little practical benefit. These findings provide both theoretical grounding and practical guidance for designing parameter-efficient prompt learning strategies.

Abstract

Adaptive prompting mechanisms have been proposed to enhance vision-language models by dynamically tailoring prompts to inputs. However, in frozen few-shot prompt learning with CLIP-style backbones, we systematically observe that adaptive gates and prompt-selection modules often collapse: they produce nearly constant outputs, contribute negligible gradient signals, and frequently fail to outperform fixed prompts. To further explore this issue, we present a systematic diagnostic study to uncover the underlying causes and conditions of adaptation failure. Through controlled experiments across datasets and multiple prompt learning architectures, we identify two recurring failure modes: gradient magnitude imbalance and gate degradation. Our findings invite a re-examination of indiscriminately adding architectural complexity in parameter-efficient learning and clarify when prompt-level adaptive gating is, and is not, effective in this regime.

Problem

• adaptive prompting
• prompt learning
• gating collapse
• vision-language models
• gradient imbalance

Innovation

• adaptive prompting
• gradient imbalance
• gate collapse
• vision-language models
• prompt learning