Look Before You Zoom: Adaptive Routing for the Resolution-Context Trade-off in Visual RAG

2026-06-20

AI Summary

This work addresses the performance degradation of existing training-free visual retrieval methods on small objects and their neglect of the trade-off between resolution and contextual information. The authors propose ViRGo, a novel framework that formulates visual retrieval as an adaptive routing problem. ViRGo leverages the built-in localization head of a vision-language model (VLM) to estimate object scale and combines it with semantic token confidence to dynamically select among global, patch-level, or attention-based retrieval pathways. Without requiring any additional training, this approach explicitly models and resolves the resolution–context trade-off, achieving a superior balance between accuracy and efficiency across multiple VQA benchmarks: it matches patch-level retrieval performance on small objects, outperforms attention-based retrieval on large objects, and reduces inference cost by avoiding unnecessary high-resolution processing.

Abstract

Vision-Language Models (VLMs) struggle as query-relevant objects become smaller. To address this, recent training-free approaches dynamically retrieve and zoom into local image regions. However, we show that indiscriminately applying retrieval ignores a critical vulnerability: the resolution-context trade-off. Patch-based zooming recovers details for small targets, but can split large objects and destroy global spatial context; attention-based retrieval better preserves large objects, but remains less reliable on tiny details; and global perception is often fastest when retrieval is unnecessary. Motivated by these failure modes, we introduce ViRGo (Visual Retrieval or Global Perception), a lightweight framework that formulates visual retrieval as an adaptive routing problem. ViRGo estimates object scale from the VLM's intrinsic localization heads during the initial forward pass and combines it with semantic token confidence to select between global perception, patch-based retrieval, and attention-based retrieval with minimal additional computation. Experiments across multiple VQA benchmarks and object-size groups show that ViRGo improves the accuracy-efficiency trade-off: it matches patch retrieval on small details, leverages attention-based retrieval for larger objects, and reduces inference time by routing to the global baseline when zooming is unnecessary.

Problem

• resolution-context trade-off
• visual retrieval
• object scale
• vision-language models
• adaptive routing

Innovation

• adaptive routing
• resolution-context trade-off
• visual retrieval
• Vision-Language Models
• object scale estimation