AdaBoosting Text Prompts for Vision-Language Models

2026-07-01

AI Summary

Existing few-shot textual prompting methods do not explicitly focus on misclassified samples, which limits their performance gains. This work proposes a Textual Prompt Boosting (TPB) framework that, for the first time, integrates the AdaBoost principle into prompt learning by treating each textual prompt classifier as a weak learner. Through iterative reweighting that emphasizes hard-to-classify examples and sequential ensemble construction, TPB builds a strong classifier while preserving task-relevant semantic cues in the textual space. The approach is model-agnostic and transferable, enabling effective prompt optimization across diverse settings. Evaluated on 11 classification benchmarks, TPB achieves substantial accuracy improvements and maintains its few-shot advantage when transferred to larger vision-language models, outperforming current state-of-the-art methods.

Abstract

The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.

Problem

• few-shot prompting
• text prompts
• Vision-Language Models
• misclassified examples
• prompt construction

Innovation

• Text Prompt Boosting
• AdaBoost
• Vision-Language Models
• Few-shot Prompting
• Prompt Ensemble