ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

2026-08-08

AI Summary

This work addresses the challenge of parameter-efficient adaptation in settings where only forward passes are permitted, rendering conventional fine-tuning infeasible. Existing zeroth-order prompt tuning methods are either restricted to unimodal settings or suffer from poor query efficiency. To overcome these limitations, we propose ZOMP—a fully forward, query-efficient zeroth-order multimodal prompt tuning framework that jointly optimizes deep prompts within both the vision and text branches of a frozen CLIP model. Our key innovations include cross-modal low-rank reparameterization to reduce search dimensionality, momentum-enhanced gradient estimation to stabilize noisy perturbations, and a budget-aware adaptive rank scheduling strategy that dynamically allocates model capacity. Under a strict query budget of 5,000 evaluations across 13 vision-language benchmarks, ZOMP significantly outperforms existing backpropagation-free methods in few-shot accuracy, query efficiency, cross-dataset transfer, and out-of-distribution generalization.

Abstract

Fine-tuning vision-language models such as CLIP typically requires backpropagation (BP) through the full model, which is infeasible when only forward-pass access is available, as is common for memory-constrained edge devices and proprietary model deployments. Prior BP-free, zeroth-order prompt-tuning methods avoid this requirement but often tune prompts in a single modality or optimize over a search space large enough that convergence requires thousands of forward passes, which is impractical under realistic query budgets. We propose ZOMP (Zeroth-Order Multimodal Prompt tuning), a query-efficient, fully forward-only method that tunes deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation. ZOMP combines three ingredients: a cross-modal low-rank reparameterization that ties the two branches through a shared factor and keeps the effective search dimensionality small, a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, and a budget-indexed rank schedule that unlocks capacity as the query budget is spent. Across 13 vision-language benchmarks under a matched 5,000-query budget, ZOMP consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency, and it generalizes better across base-to-new, cross-dataset transfer, and out-of-distribution settings. Our results show that jointly exploiting multimodality and low-rank structure is an effective route to practical, query-efficient BP-free prompt tuning.

Problem

• zeroth-order optimization
• prompt tuning
• vision-language models
• query efficiency
• multimodal learning

Innovation

• zeroth-order optimization
• multimodal prompt tuning
• low-rank reparameterization
• query-efficient learning
• vision-language models