AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition

2025-12-03

AI Summary

Vision-language models (VLMs) suffer from high computational overhead in visual question answering (VQA) due to redundant visual tokens. Existing efficient methods employ fixed-ratio token compression, lacking sample-level adaptivity. To address this, we propose an adaptive visual token acquisition mechanism: (1) a human-inspired active vision paradigm enabling the model to autonomously decide whether to invoke bounding-box tools for localizing salient regions; (2) a coarse-to-fine dynamic information acquisition strategy that retrieves visual details on-demand; and (3) a Decoupled Turn-wise Policy Optimization (DTPO) framework, which separately optimizes tool invocation and answer generation under reinforcement learning. Experiments demonstrate that our method significantly outperforms state-of-the-art efficient VLMs across multiple VQA benchmarks—achieving accuracy gains while reducing visual tokens by 30–60%.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.

Problem

• Reduces computational overhead in vision-language models by adaptive visual token acquisition
• Enables models to autonomously determine minimum visual tokens per sample through coarse-to-fine processing
• Balances accuracy and efficiency using reinforcement learning with decoupled policy optimization

Innovation

• Adaptive visual token acquisition via coarse-to-fine approach
• Reinforcement learning with Decoupled Turn Policy Optimization (DTPO)
• Selective bounding box tool for cropping key image regions