SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

2026-07-06

AI Summary

This work addresses the poor generalization of vision-language models (VLMs) in prompt learning, often caused by overfitting to seen data. To mitigate this, the authors propose SAMPLe—the first optimizer that integrates Sharpness-Aware Minimization (SAM) into prompt learning. By jointly leveraging local curvature and gradient information of the loss landscape, SAMPLe dynamically balances exploration and exploitation during optimization. The method is model-agnostic, plug-and-play, and compatible with mainstream prompt-learning frameworks such as CoOp, CoCoOp, and MaPLe. Extensive experiments demonstrate that SAMPLe consistently outperforms existing optimization strategies across multiple downstream tasks and out-of-distribution settings, significantly enhancing generalization while maintaining competitive task performance.

Abstract

Pre-trained Vision-Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for prompt learning.

Problem

• prompt learning
• vision-language models
• generalization
• overfitting
• performance-generalization dilemma

Innovation

• Prompt Learning
• Sharpness-Aware Minimization
• Vision-Language Models
• Generalization
• Optimization