MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning

2026-07-14

AI Summary

This work addresses the limitation of current vision-language models, which achieve high Top-K accuracy in few-shot classification but suffer from suboptimal Top-1 performance due to difficulties in distinguishing visually similar categories. To overcome this, the authors propose a coarse-to-fine fine-tuning strategy: first selecting Top-K semantically plausible candidate classes as anchors, then refining visual representations through a cross-modal quantum learning mechanism. Innovatively integrating quantum computation into multimodal adapters, the method leverages quantum state encoding, superposition, and entanglement to model high-order cross-modal interactions, thereby enhancing fine-grained discriminability while maintaining parameter efficiency. Experiments demonstrate that the approach significantly improves Top-1 accuracy across 15 benchmark datasets and reduces the number of trainable parameters.

Abstract

Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate categories and thus achieve high Top-K accuracy. However, they often struggle with fine-grained discrimination among visually similar categories, resulting in unsatisfactory Top-1 performance, as shown in Figure 1. Existing studies on VLM adapters generally focus on global alignment between visual and textual representations in the feature space, but fail to exploit semantically similar categories to refine fine-grained visual representations. Based on these observations, we propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation, termed the Multi-Modal Quantum Adapter (MQAdapter). Specifically, MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors. It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors. The core of this mechanism is the encoding of visual and textual features into quantum states. By leveraging quantum entanglement and superposition in a high-dimensional Hilbert space, MQAdapter effectively models higher-order cross-modal interactions, producing more discriminative representations than traditional Euclidean adapters. MQAdapter is parameter-efficient and can be integrated with various existing fine-tuning algorithms to achieve further performance gains. Evaluations on 15 datasets demonstrate the effectiveness of MQAdapter while requiring fewer trainable parameters.

Problem

• Vision-Language Models
• few-shot classification
• fine-grained discrimination
• Top-1 accuracy
• visual similarity

Innovation

• Quantum Adapter
• Coarse-to-Fine Fine-tuning
• Cross-Modal Quantum Learning
• Vision-Language Models
• Few-Shot Learning