MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

2026-07-13

AI Summary

Existing vision-based tool-using agents lack a unified, realistic, and state-aware evaluation benchmark, making it difficult to assess their ability to integrate visual understanding with tool execution in multi-image, multi-turn interactions. This work proposes the first unified evaluation framework for vision-guided tool use, featuring a stateful execution environment spanning 16 domains and supporting over 500 tools. Diverse visually grounded tasks are generated through an information-flow-guided automated scene synthesis pipeline coupled with a multi-stage filtering mechanism. Benchmarking 12 leading models reveals that even the strongest model achieves a success rate below 50%, with 53% of failures attributable to errors in image information extraction—indicating that the current performance bottleneck has shifted from planning capability to visual perception accuracy.

Abstract

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

Problem

• visual tool-calling
• visually grounded agents
• information extraction
• multi-turn tasks
• visual precision

Innovation

• visual tool-calling
• stateful execution environment
• automated scenario generation
• information-flow-guided planning
• visual grounding