Adaptive Prompt: Unlocking the Power of Visual Prompt Tuning

2025-01-31

AI Summary

Visual Prompt Tuning (VPT) suffers from limited generalization and parameter efficiency due to its fixed, task-agnostic prompt representations, which hinder adaptation to diverse downstream tasks. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), the first method to model visual prompts as input-dependent, learnable functions—enabling dynamic prompt generation conditioned on input features. Built upon the Transformer architecture, VAPT introduces a lightweight parametric mapping module that transforms input embeddings into task-specific prompts. We theoretically establish its optimal sample efficiency under standard assumptions. Extensive experiments on VTAB-1K and FGVC benchmarks demonstrate that VAPT outperforms full fine-tuning by 7.34% and 1.04%, respectively, and significantly surpasses VPT across all settings. Crucially, VAPT achieves these gains with only a negligible number of additional trainable parameters—delivering both superior performance and exceptional parameter efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Visual Prompt Tuning (VPT) has recently emerged as a powerful method for adapting pre-trained vision models to downstream tasks. By introducing learnable prompt tokens as task-specific instructions, VPT effectively guides pre-trained transformer models with minimal overhead. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on recent insights into the connection between mixture of experts and prompt-based approaches, we identify a key limitation in VPT: the restricted functional expressiveness in prompt formulation. To address this limitation, we propose Visual Adaptive Prompt Tuning (VAPT), a new generation of prompts that redefines prompts as adaptive functions of the input. Our theoretical analysis shows that this simple yet intuitive approach achieves optimal sample efficiency. Empirical results on VTAB-1K and FGVC further demonstrate VAPT's effectiveness, with performance gains of 7.34% and 1.04% over fully fine-tuning baselines, respectively. Notably, VAPT also surpasses VPT by a substantial margin while using fewer parameters. These results highlight both the effectiveness and efficiency of our method and pave the way for future research to explore the potential of adaptive prompts.

Problem

• Visual Prompt Tuning
• Limited Expressiveness
• Image Recognition Models

Innovation

• Visual Adaptive Prompt Tuning
• Dynamic Adjustment
• Efficient Learning