Indexing Multimodal Language Models for Large-scale Image Retrieval

2026-04-14

AI Summary

This work investigates the potential of multimodal large language models (MLLMs) to perform purely visual tasks without any training, with a focus on instance-level similarity assessment in large-scale image retrieval. The authors propose a zero-shot reranking method that feeds image pairs into an MLLM and converts its next-token prediction probabilities into similarity scores. Coupled with a memory-efficient indexing mechanism, this approach enables scalable top-k reranking. Notably, it is the first to directly apply MLLMs—without fine-tuning or task-specific architectures—to training-agnostic large-scale image retrieval reranking. The method outperforms specialized rerankers trained on non-native domains across multiple benchmarks and demonstrates superior robustness in challenging scenarios such as cluttered backgrounds, occlusions, and small objects.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-$k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval.

Problem

• Multimodal Large Language Models
• Image Retrieval
• Zero-shot Re-ranking
• Instance-level Similarity
• Large-scale Vision Tasks

Innovation

• Multimodal Large Language Models
• zero-shot image retrieval
• training-free re-ranking
• similarity estimation
• large-scale indexing