U-shaped Multi-granularity Learning for Vision-Language Models

2026-07-16

AI Summary

This work addresses the granularity dilemma in vision-language prompt learning, where global prompts suffer from coarse semantics and local prompts lack contextual coherence, thereby limiting cross-task generalization. Inspired by U-Net, we propose UPrompt—the first U-shaped multi-granularity prompt learning framework for vision-language models—that concurrently constructs multi-granular representations for both visual and textual modalities. By integrating coarse-to-fine cascaded enhancement with fine-to-coarse hierarchical supervision, UPrompt achieves cross-scale semantic consistency and detail refinement. Its symmetric encoder-decoder architecture with cross-level connections significantly boosts generalization, outperforming state-of-the-art methods across 17 benchmarks: it surpasses MAMET and VPKE by 4.1 and 7.3 in rSum on MSCOCO, exceeds CoCoA-Mix by 5.09% in base-to-novel class performance, and incurs only one-third the computational overhead of PSRC.

Abstract

The prompt learning paradigm for vision-language models is effective yet faces a granularity dilemma: global prompts lack fine-grained semantic awareness, while local prompts ignore contextual associations, limiting cross-task generalization. This dilemma exists in dense prediction tasks. Inspired by U-Net, which unifies multi-level representations across granularities, we propose UPrompt, a U-shaped multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global context to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, surpasses CoCoA-Mix by 5.09% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).

Problem

• prompt learning
• granularity dilemma
• vision-language models
• dense prediction
• cross-task generalization

Innovation

• U-shaped architecture
• multi-granularity learning
• prompt learning
• vision-language models
• cross-level feature fusion