TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

2026-09-14

AI Summary

研究通过创建TwinICL基准解决多模态在上下文学习中的性能评估问题，使用配对反事实方法来诊断并提升多模态模型的表现。

Abstract

In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.

Problem

• in-context learning
• multimodal
• benchmark

Innovation

• TwinICL
• multimodal ICL
• procedurally generated benchmark
• counterfactuals
• visual access