VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

2026-03-24

AI Summary

This work addresses the information bottleneck introduced by visual token compression in existing large vision-language models, which compromises fine-grained understanding. To overcome this limitation, the authors propose a sparse vision-language interaction mechanism that abandons conventional token compression entirely, preserving full high-resolution visual details while dynamically activating self-attention layers only at critical spatial locations to refine representations. The approach integrates cross-attention to maintain global visual context and employs a lightweight policy network that adaptively allocates computational resources based on input complexity. Experimental results demonstrate that the method matches or surpasses state-of-the-art performance across multiple benchmarks while significantly reducing computational overhead, with particularly pronounced gains on tasks sensitive to high-resolution visual details.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.

Problem

• Large Vision-Language Models
• visual token reduction
• information bottleneck
• computational efficiency
• fine-grained visual understanding

Innovation

• sparse attention
• dynamic selection
• vision-language interaction
• computational efficiency
• high-resolution reasoning