Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning

2026-08-10

AI Summary

This work investigates whether the performance gains in visual reasoning stem from the actual images returned by tool calls or from the structured textual prompts they provide. To isolate the contribution of textual scaffolding, the authors propose TextCall, which retains the structured text (e.g., tool names, coordinates, object descriptions) generated by tool calls while replacing the actual images with placeholders. Through LoRA fine-tuning, full-parameter fine-tuning, and reinforcement learning—combined with blind evaluation, gain decomposition, and attention analysis—the study provides the first systematic validation of the “tool-call textual scaffolding hypothesis.” Experiments demonstrate that TextCall matches or even surpasses methods using full image feedback across diverse training settings, reduces inference latency by 29–46%, eliminates the need for real API calls, and still preserves effective tool-use behavior.

Abstract

Tool-augmented vision-language models increasingly "think with images": they call crop, zoom, or code tools and reason over the returned pixels. However, recent work using blind tests, gain decompositions, and attention analyses has shown that returned images contribute little, raising the question: if pixels do not carry the gain, what does? We hypothesize that the load-bearing signal is the structured text emitted before any returned pixel arrives: tool name, coordinates, target description, and intent. This textual scaffold encodes where to look and what to find. We introduce TextCall (call-but-no-return) to test this: it keeps the scaffold but replaces returned images with the text placeholder [Image output skipped]. Three studies support the hypothesis. (i) Non-necessity of returned pixels: across LoRA, full fine-tuning, and RL, TextCall matches or exceeds full thinking-with-images; under RL it preserves tool use at the reported checkpoint, avoiding the failure mode where, under matched settings, seeing the returned image causes the model to stop calling tools and answer directly. (ii) Sufficiency of the scaffold: on matched training queries, scaffold-only input yields equivalent accuracy to returned-image input. (iii) Component specificity: decomposing the scaffold into reasoning text and spatial code shows both components contribute, with the dominant one varying by task. Together these results support the Tool-Call Scaffold Hypothesis: in current thinking-with-images distributions, the active signal is the structured text emitted at tool-call time; the returned image is a redundant carrier. TextCall preserves accuracy while reducing latency by 29-46% and eliminating tool-execution API calls. Our claims hold for current thinking-with-images benchmarks; constructing tasks where pixels are genuinely load-bearing remains an open direction.

Problem

• tool-augmented vision-language models
• visual reasoning
• tool calls
• text scaffold
• returned pixels

Innovation

• tool-call scaffold
• visual reasoning
• vision-language models
• TextCall
• structured text