Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?

2025-08-07

AI Summary

This work addresses the limited fine-grained text localization and reasoning capabilities of multimodal large language models (MLLMs) on complex documents. To this end, we introduce NiM—the first benchmark explicitly designed for precise, fine-grained text localization in document images—and propose Spot-IT, a novel method inspired by human reading behavior. Spot-IT integrates intelligent image patch sampling, visual token pruning, and Gaussian-weighted attention to enhance model sensitivity to critical textual details (e.g., nutritional values in menus or disclaimers in newspapers). Experiments reveal that state-of-the-art MLLMs achieve only modest performance on NiM, whereas Spot-IT significantly improves localization accuracy—particularly on layout-complex, clutter-rich documents. This work establishes a new, challenging benchmark for document-level fine-grained understanding and provides an interpretable, computationally efficient methodological framework.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

While Multi-modal Large Language Models (MLLMs) have shown impressive capabilities in document understanding tasks, their ability to locate and reason about fine-grained details within complex documents remains understudied. Consider searching a restaurant menu for a specific nutritional detail or identifying a disclaimer in a lengthy newspaper article tasks that demand careful attention to small but significant details within a broader narrative, akin to Finding Needles in Images (NiM). To address this gap, we introduce NiM, a carefully curated benchmark spanning diverse real-world documents including newspapers, menus, and lecture images, specifically designed to evaluate MLLMs' capability in these intricate tasks. Building on this, we further propose Spot-IT, a simple yet effective approach that enhances MLLMs capability through intelligent patch selection and Gaussian attention, motivated from how humans zoom and focus when searching documents. Our extensive experiments reveal both the capabilities and limitations of current MLLMs in handling fine-grained document understanding tasks, while demonstrating the effectiveness of our approach. Spot-IT achieves significant improvements over baseline methods, particularly in scenarios requiring precise detail extraction from complex layouts.

Problem

• Evaluating MLLMs' ability to locate fine details in documents
• Introducing NiM benchmark for real-world document detail analysis
• Proposing Spot-IT to enhance MLLMs' fine-grained detail extraction

Innovation

• Introduces NiM benchmark for fine-grained detail evaluation
• Proposes Spot-IT with patch selection and Gaussian attention
• Enhances MLLMs' precision in complex document understanding