LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models

2026-06-15

AI Summary

This work addresses the challenge that multimodal large language models struggle to precisely locate critical local visual evidence amidst redundant information in fine-grained perception tasks. To this end, the authors propose LOCUS, a novel training framework that, for the first time, incorporates verifiable local cue searching as a proxy task during training. Specifically, localized image crops are generated as visual cues, and an IoU-based reward guides the model to recover their spatial locations within the full image, thereby internalizing the ability to search for task-relevant local evidence. Notably, LOCUS enhances the model’s focus on relevant regions without modifying the inference interface. Experiments demonstrate that LOCUS significantly improves performance across multiple benchmarks—including fine-grained visual understanding, hallucination suppression, and general multimodal reasoning—while preserving broad multimodal capabilities.

Abstract

Multimodal Large Language Models (MLLMs) remain unreliable on fine-grained visual perception, even when high-resolution inputs preserve the necessary local details. We identify this limitation as visual context rot: decisive evidence may exist in the full image, yet fail to be reliably selected and used amid redundant visual context. We propose LOCUS (LOcal visual CUe Search), a training framework that teaches MLLMs to internalize local evidence search through a verifiable proxy task. During training, LOCUS provides a local crop as a visual cue and optimizes the model to recover its spatial support in the full image using an IoU-based reward. The visual cue is used only during training, leaving the standard image-question inference interface unchanged. Experiments across fine-grained perception, hallucination, general understanding, and reasoning benchmarks show that LOCUS improves localization-sensitive visual understanding while preserving broad capabilities. Attention analyses further indicate stronger focus on task-relevant evidence regions, suggesting that training-time visual cue search provides an effective route to internalized fine-grained evidence selection.

Problem

• fine-grained visual perception
• multimodal large language models
• visual context rot
• local visual cues
• evidence selection

Innovation

• fine-grained perception
• visual context rot
• local visual cue
• IoU-based reward
• multimodal large language models