Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools

2026-07-13

AI Summary

Current vision-language models suffer from high computational overhead and inference latency in fine-grained perception tasks due to frequent external tool invocations and redundant image encodings. This work proposes an implicit visual tool paradigm that integrates tool usage into the training objective through a self-regulation mechanism, enabling the model to adaptively balance internal knowledge and external tools. The approach employs a two-stage training strategy: first, chain-of-thought supervised fine-tuning with structured tool slots, followed by reward-driven alignment guided by Net Tool Gain (NTG) to minimize redundant calls and enable effective knowledge routing. Experiments demonstrate that the method achieves state-of-the-art performance on fine-grained visual tasks while preserving general reasoning capabilities and significantly improving inference efficiency.

Abstract

Recent multimodal large language models (MLLMs) have made remarkable progress on fine-grained perception tasks under the "Thinking with Images" (TwI) paradigm by iteratively performing various visual tool operations. However, this paradigm relies heavily on frequent external tool calls and repeated image re-encoding, which leads to substantial computational overhead and inference latency. To address these issues, we propose Beyond the Eye (BEE), a novel implicit visual tool paradigm centered on self-regulated capability. BEE directly incorporates visual tool invocation behaviors into the training objective and encourages the model to develop a self-regulated invocation mechanism. This design enables the model to adaptively balance internal knowledge and implicit tools, avoiding redundant tool usage while substantially reducing inference latency. Specifically, BEE involves a two-stage training process: (1) Formalized Chain-of-Thought (CoT) Supervised Fine-tuning (SFT). We construct CoT trajectories with structured tool slots and mixed invocation states. This stage activates the model's implicit tool representations and adaptive switching capability. (2) Self-regulated Reward-Driven Alignment. To address redundant tool usage caused by ambiguous cognitive boundaries, we first introduce the Net Tool Gain (NTG) metric to quantify this phenomenon. Based on this observation, we further propose a self-regulated reward mechanism. This mechanism penalizes ineffective tool dependency and encourages the model to perform knowledge routing, ensuring that implicit tools are invoked only when the model's internal knowledge is insufficient. BEE achieves state-of-the-art performance in fine-grained visual perception while remaining competitive in general reasoning tasks and achieving substantial gains in inference efficiency.

Problem

• multimodal reasoning
• visual tools
• inference latency
• computational overhead
• fine-grained perception

Innovation

• implicit visual tools
• self-regulated reasoning
• multimodal large language models
• Net Tool Gain
• efficient inference