Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

2025-08-01

AI Summary

This paper addresses the modality imbalance problem in vision-language prompt tuning (PT), where raw images contain substantially richer contextual information than textual descriptions—causing cross-modal alignment to favor background regions and undermining foreground object focus. To tackle this, we propose a “disentangle-then-align” framework. Our method comprises three key components: (1) foreground-background visual feature disentanglement guided by coarse- and fine-grained segmentation cues; (2) background-category–guided textual alignment using handcrafted background class prompts; and (3) a vision-based pull-push regularization that explicitly enhances attention to target regions. To our knowledge, this is the first work to systematically identify and mitigate modality bias in PT. Extensive experiments demonstrate significant improvements over state-of-the-art methods across three critical settings: few-shot learning, base-to-novel generalization, and data-efficient learning—achieving new SOTA results on mainstream benchmarks.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Prompt tuning (PT), as an emerging resource-efficient fine-tuning paradigm, has showcased remarkable effectiveness in improving the task-specific transferability of vision-language models. This paper delves into a previously overlooked information asymmetry issue in PT, where the visual modality mostly conveys more context than the object-oriented textual modality. Correspondingly, coarsely aligning these two modalities could result in the biased attention, driving the model to merely focus on the context area. To address this, we propose DAPT, an effective PT framework based on an intuitive decouple-before-align concept. First, we propose to explicitly decouple the visual modality into the foreground and background representation via exploiting coarse-and-fine visual segmenting cues, and then both of these decoupled patterns are aligned with the original foreground texts and the hand-crafted background classes, thereby symmetrically strengthening the modal alignment. To further enhance the visual concentration, we propose a visual pull-push regularization tailored for the foreground-background patterns, directing the original visual representation towards unbiased attention on the region-of-interest object. We demonstrate the power of architecture-free DAPT through few-shot learning, base-to-novel generalization, and data-efficient learning, all of which yield superior performance across prevailing benchmarks. Our code will be released at https://github.com/Ferenas/DAPT.

Problem

• Addresses visual-textual information asymmetry in prompt tuning
• Proposes decoupling visual modality into foreground and background
• Enhances modal alignment via pull-push regularization for unbiased attention

Innovation

• Decouples visual modality into foreground-background
• Aligns decoupled patterns with foreground-background texts
• Uses pull-push regularization for unbiased attention