Problem
Research questions and friction points this paper is trying to address.
Prompt Tuning
CLIP
Patch-Grid Phase Sensitivity
Vision-Language Models
Spatial Sampling
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
🤖 AI Summary
This study addresses the phase sensitivity of frozen ViT patch grids in CLIP prompt tuning by proposing PPOM, a training-free inference operator. Introducing patch phase orbit marginalization, this method treats phase shifts as nuisance variables and achieves unbiased phase integration through symmetric translation and anti-family pairing techniques, thereby establishing a deterministic interface between prompt adaptation and grid sensitivity. Extensive experiments demonstrate that PPOM significantly enhances performance across diverse prompt learning frameworks without retraining. Furthermore, the approach effectively improves model generalizability and multi-host adaptability, validating its broad applicability as a universal enhancement for vision-language models.
📝 Abstract
Prompt tuning adapts CLIP-based vision-language models with few trainable parameters, yet its predictions remain sensitive to the spatial sampling imposed by a frozen vision transformer. In particular, non-overlapping patch tokenization makes predictions depend on the alignment (phase) between image and the patch lattice. To reduce prediction sensitivity to patch-grid alignment, we introduce Patch-Phase Orbit Marginalization (PPOM), a training-free inference operator that treats phase shift as a nuisance variable. Given a patch stride, PPOM evaluates the identity view and reflection-padded translations, pairs opposite shifts into horizontal, vertical, and diagonal antithetic families, and assigns equal mass to these families and the identity prediction to avoid view-count bias during phase integration. In summary, PPOM provides a deterministic interface between prompt adaptation and patch-grid sensitivity. Across multiple prompt-learning hosts, PPOM improves host performance without re-training.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Patch-Phase Orbit Marginalization
Training-free Inference
CLIP Prompt Tuning
Patch-grid Sensitivity
Antithetic Families
🔎 Similar Papers
No similar papers found.
