OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories

2026-08-09

AI Summary

This work addresses a critical limitation in existing visual tool-learning methods, which rely on teacher-generated trajectories where tool invocations often lack causal relevance to the final answer, leading models to learn spurious correlations rather than meaningful reasoning. To overcome this, the authors propose a three-stage open framework that integrates difficulty-based filtering, domain-adaptive trajectory synthesis, and causal utility validation to construct high-quality trajectories that are not only outcome-correct but also exhibit genuine causal contributions from tool usage. Notably, this is the first approach to adopt causal utility as a trajectory selection criterion, ensuring that tools substantively aid reasoning. The resulting OpenVisTool-42K dataset, comprising 42K samples, serves as both a training resource and evaluation benchmark. Fine-tuning multimodal models (4B–27B parameters) on this data consistently enhances tool-use proficiency and demonstrates strong out-of-domain generalization, with large models approaching the performance of leading closed-source systems.

Abstract

Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding. The prevailing recipe learns this capability from teacher-generated trajectories filtered for answer correctness, implicitly assuming that every successful demonstration provides effective supervision. We argue this assumption is flawed: a strong teacher often reaches the correct answer without needing its tool calls, and imitating such trajectories teaches a student that tool calls accompany correct answers, not that tool observations ground them. We present OpenVisTool, an open framework for constructing instructive visual tool-use trajectories that provide effective supervision for tool learning. The key insight is that a trajectory should be retained only if its answer is correct (outcome validity) and its tool observations causally contribute to that answer (causal utility). The framework operates in three stages: difficulty screening to select queries that are not reliably answerable without tools, domain-specific trajectory synthesis to elicit coherent tool-use trajectories, and supervision verification to jointly test both conditions. Rather than encouraging models to imitate tool calls, the resulting supervision teaches when and how visual evidence should be acquired. Using this framework, we construct OpenVisTool-42K, a dataset spanning five visual reasoning domains, together with OpenVisTool-Bench, a benchmark covering the same domains. Across four backbones (4B-27B), fine-tuning on OpenVisTool-42K consistently improves visual tool-use performance and yields gains on two out-of-distribution benchmarks; the larger models approach leading closed-source systems. The evidence suggests that effective visual tool use is learned from causally grounded supervision rather than tool-calling patterns.

Problem

• visual tool use
• causal utility
• supervision quality
• multimodal agents
• trajectory synthesis

Innovation

• causal utility
• visual tool use
• instructive trajectories
• outcome validity
• multimodal agents