Large Multimodal Models as General In-Context Classifiers

2026-02-26

AI Summary

This work addresses the performance bottleneck of large multimodal models (LMMs) in both open- and closed-world classification tasks, which arises from imperfect contextual information. To this end, the authors propose CIRCLE, a training-free in-context learning framework that iteratively refines pseudo-labels of context examples to enhance classification robustness under few-shot and even zero-shot settings. Experimental results demonstrate that CIRCLE significantly outperforms contrastive vision-language models and their adapter-based variants across multiple benchmark datasets. This study provides the first systematic validation of LMMs as versatile, training-free classifiers, establishing their feasibility and superiority in general-purpose classification without task-specific fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.

Problem

• Large Multimodal Models
• In-Context Learning
• Open-World Classification
• Vision-Language Models
• Zero-Shot Classification

Innovation

• In-context Learning
• Large Multimodal Models
• Open-world Classification
• CIRCLE
• Vision-Language Models