Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering

2025-06-07

AI Summary

In few-shot visual question answering (VQA), large multimodal models (LMMs) exhibit unstable and non-monotonic performance under in-context learning (ICL), primarily due to redundant image embeddings interfering with task-relevant feature extraction. Method: We propose a meta-adaptive soft prompt distillation framework that, for the first time, distills task-specific visual features into lightweight, test-time adaptable soft prompts. A plug-and-play attention mapping module enables joint optimization of visual and linguistic representations. Built upon LLaVA-v1.5, our approach integrates meta-learning, soft prompt tuning, attention modeling, and gradient-efficient fine-tuning. Contribution/Results: On VL-ICL Bench, our method significantly outperforms standard ICL and state-of-the-art prompt tuning baselines. It demonstrates strong robustness to image perturbations and substantially enhances few-shot task induction and cross-question-type reasoning capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, is inconsistent and does not always improve monotonically with increasing examples. We hypothesize that this occurs due to the LMM being overwhelmed by additional information present in the image embeddings, which is not required for the downstream task. To address this, we propose a meta-learning approach that provides an alternative for inducing few-shot capabilities in LMMs, using a fixed set of soft prompts that are distilled from task-relevant image features and can be adapted at test time using a few examples. To facilitate this distillation, we introduce an attention-mapper module that can be easily integrated with the popular LLaVA v1.5 architecture and is jointly learned with soft prompts, enabling task adaptation in LMMs under low-data regimes with just a few gradient steps. Evaluation on the VL-ICL Bench shows that our method consistently outperforms ICL and related prompt-tuning approaches, even under image perturbations, improving task induction and reasoning across visual question answering tasks.

Problem

• Inconsistent ICL performance in smaller LMMs
• Overwhelming image embeddings hinder downstream tasks
• Few-shot adaptation needs efficient meta-learning solutions

Innovation

• Meta-learning for few-shot LMM adaptation
• Attention-mapper module for feature distillation
• Soft prompts from task-relevant image features