Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models

2025-07-11

AI Summary

Current vision-language models (VLMs) face two key bottlenecks in prompt learning: inadequate semantic modeling of unseen class embeddings and coarse-grained cross-modal alignment restricted to encoder top-layer outputs, which compromises topological consistency. To address these, we propose MuGCP—a novel multimodal prompting framework. First, it leverages multimodal large language models (MLLMs) to dynamically generate semantics-conditioned prompts. Second, it introduces an attention-based mutual guidance module that enables fine-grained, intermediate-layer interaction between visual and textual encoders. Third, it incorporates a multi-prompt fusion mechanism to jointly enhance class-level representation learning and instance-aware discrimination. Evaluated across 14 benchmark datasets, MuGCP achieves significant improvements over state-of-the-art methods, demonstrating superior generalization in zero-shot classification and fine-grained recognition, as well as stronger cross-modal alignment fidelity and topological coherence.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Prompt learning facilitates the efficient adaptation of Vision-Language Models (VLMs) to various downstream tasks. However, it faces two significant challenges: (1) inadequate modeling of class embedding distributions for unseen instances, leading to suboptimal generalization on novel classes; (2) prevailing methodologies predominantly confine cross-modal alignment to the final output layer of vision and text encoders, which fundamentally limits their capacity to preserve topological consistency with pre-trained multi-modal embedding spaces. To this end, we introduce MuGCP (Multi-modal Mutual-Guidance Conditional Prompt Learning), a novel paradigm designed for conditional prompt generation. MuGCP leverages Multi-modal Large Language Models (MLLMs) as conditional prompt learners to adaptively generate Semantic Conditional Prompts (SCP) that incorporate rich, fine-grained high-level semantic knowledge for image instances. To ensure effective alignment and interaction across the multi-modal space of Vision-Language Models (VLMs), we introduce the Attention Mutual-Guidance (AMG) module, which facilitates interactions between visual and semantic information. Through mutual guidance, the AMG module generates Visual Conditional Prompts (VCP), enhancing the model's performance in multi-modal tasks. Additionally, we present a Multi-Prompt Fusion (MPF) mechanism that integrates SCP and VCP with contextual prompts, ensuring seamless coordination among the different prompts and enhancing the modeling of class embeddings and instance-specific knowledge. Our MuGCP outperforms existing state-of-the-art methods on 14 different datasets. The code will be made available after publication.

Problem

• Improve generalization for novel classes in VLMs
• Enhance cross-modal alignment beyond final encoder layers
• Integrate multi-modal prompts for better semantic consistency

Innovation

• Multi-modal Mutual-Guidance Conditional Prompt Learning
• Attention Mutual-Guidance module for cross-modal alignment
• Multi-Prompt Fusion mechanism integrating SCP and VCP