Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding

2026-06-14

AI Summary

This work addresses the challenge faced by multimodal large language models in simultaneously preserving fine-grained details and global context when processing high-resolution images. Existing training-free approaches rely on uniform scaling or cropping strategies, often resulting in computational redundancy for simple queries and critical information loss for complex ones. To overcome this limitation, the authors propose LazyMCoT, a novel framework that achieves dynamic visual grounding without any additional training. It leverages statistical properties of the initial token to estimate query difficulty and employs an adaptive routing mechanism to allocate computational resources accordingly. For challenging samples, LazyMCoT integrates an internal cross-modal attention module with external vision experts through a two-stage refinement process, further enhanced by conformal calibration to guarantee recall. The method matches or exceeds the performance of trainable baselines across multiple benchmarks while significantly reducing average inference latency and improving accuracy.

Abstract

While Multimodal Large Language Models (MLLMs) excel in cross-modal reasoning, they often struggle to perceive fine-grained details in complex high-resolution images. Recent training-free methods address this through image scaling and localized cropping. However, applying these manipulations indiscriminately introduces computational redundancy for simple queries and can degrade accuracy by truncating essential global context or introducing irrelevant background noise. To this end, we propose LazyMCoT, a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty. The framework features an Adaptive Routing mechanism that evaluates predictive uncertainty using first-token statistics from a single forward pass. This efficiently bypasses confident cases while ensuring the recall of difficult samples via conformal calibration. For these challenging cases, a Collaborative Grounding module integrates the inherent cross-modal attention of the model with an external visual expert through a two-stage refinement process. This refinement process generates a precise localized display to recover small or occluded targets. Extensive experiments across diverse benchmarks demonstrate that LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency. Our code is availble at https://github.com/TencentBAC/LazyMCoT.

Problem

• visual grounding
• multimodal large language models
• high-resolution images
• computational redundancy
• fine-grained details

Innovation

• Adaptive Routing
• Collaborative Grounding
• Training-Free Visual Grounding
• Conformal Calibration
• Multimodal Large Language Models