MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning

2026-08-12

AI Summary

This work addresses the sensitivity of multimodal large language models to the quality and coverage of demonstration examples in few-shot in-context learning, as well as their limited ability to leverage unlabeled multimodal data. To overcome these challenges, the authors propose MAG, a novel framework that introduces manifold-guided semi-supervised propagation into multimodal in-context learning for the first time. MAG constructs a multimodal graph and operates in two stages: first, it propagates text-dominant relevance signals to identify high-impact unlabeled samples and generate pseudo-labels; second, it jointly leverages visual and textual modalities to select high-quality demonstrations. Evaluated across eight benchmarks, MAG significantly outperforms strong baselines, achieving consistent performance gains under label-scarce conditions with a limited pseudo-labeling budget.

Abstract

Few-shot in-context learning (ICL) with multi-modal large language models (MLLMs) enables task adaptation without parameter updates, but its performance is highly sensitive to the quality and coverage of the selected demonstrations. While unlabeled multi-modal data is abundant, it remains elusive how to exploit them for ICL. We propose MAG (MAnifold-Guided semi-supervised in-context demonstra- tion selection), an efficient framework that leverages unlabeled data to improve multi-modal ICL. MAG formulates demonstration selection as a semi-supervised propagation problem on a multi-modal graph and adopts a two-stage strategy: (i) relevance score propagation identifies a compact set of high-impact unlabeled samples for pseudo-labeling, reducing MLLM inference cost; (ii) multi-modal relevance is used to select the final demonstrations. We show that textual represen- tations are more effective for relevance propagation, while both visual and textual modalities are crucial for high-quality demonstration selection. Experiments on eight multi-modal benchmarks demonstrate that MAG consistently outperforms strong baselines in label-scarce regimes, achieving significant gains with a limited pseudo-labeling budget.

Problem

• in-context learning
• multi-modal learning
• semi-supervised learning
• demonstration selection
• few-shot learning

Innovation

• semi-supervised learning
• in-context learning
• multi-modal graph
• demonstration selection
• manifold propagation