ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

2026-08-21

AI Summary

本文提出Energy-Shaped Visual Prompting (ES-VP)，通过低秩初始化和能量引导动态适应生成特定图像的提示，以较少参数实现优于现有方法的性能，解决视觉提示中的灵活性与效率之间的权衡问题。

Abstract

Visual prompting (VP) has emerged as a parameter-efficient method for adapting pre-trained models to downstream tasks. However, existing approaches encounter a trade-off between flexibility and efficiency. Some methods apply a fixed prompt to all images, ignoring individual image characteristics, while others introduce auxiliary networks to generate diverse prompts. Although the latter can improve performance, it also significantly increases parameter usage and the potential for overfitting to specific datasets. Furthermore, the auxiliary networks, combined with inherent biases in pre-trained models, limit scalability and generalization. In this paper, we propose Energy-Shaped Visual Prompting (ES-VP), a novel approach that generates image-specific prompts using low-rank initialization and energy-guided dynamic adaptation, achieving superior performance with fewer parameters compared to single-prompt methods. ES-VP directly utilizes the pre-trained model for adaptive prompt generation, ensuring both parameter efficiency and improved generalization. Extensive experiments conducted on five architectures across fifteen datasets demonstrate that ES-VP consistently outperforms current state-of-the-art (SOTA) single and diverse VP methods. For instance, using the CLIP architecture across four datasets, ES-VP outperforms the SOTA method DAM-VP by an average of 2.6\% in accuracy while utilizing 590$\times$ fewer VP parameters, thereby establishing a new benchmark for efficient and generalizable model adaptation.

Problem

• visual prompting
• parameter efficiency
• model adaptation
• overfitting
• generalization

Innovation

• Energy-Shaped Visual Prompting
• low-rank initialization
• energy-guided dynamic adaptation
• parameter efficiency
• generalization