VTOS: Learning to Orchestrate Vision Tools by Co-Searching Solutions and Observers

2026-06-17

AI Summary

Existing visual programming approaches rely on fixed tool pipelines, limiting their generalization in complex scenarios such as dense object counting, occlusion, small targets, and domain shifts. This work proposes VTOS, a novel framework that introduces, for the first time, a co-search mechanism between solution programs and observer programs. It jointly optimizes executable visual tool compositions—such as Grounding DINO, SAM, and NMS—and diagnostic observers that identify failure modes. Feedback from executions is accumulated in a shared VisionThoughts knowledge base to guide subsequent searches, enabling dynamic tool orchestration and a closed-loop error feedback system. Evaluated on LVIS-Count for dense counting and PlantSeg-OOD for zero-shot disease segmentation, VTOS significantly outperforms static pipelines and existing visual programming agents, demonstrating superior robustness and generalization in challenging visual tasks.

Abstract

Vision foundation tools such as open-vocabulary detectors, segmentation models, and post-processing operators are powerful building blocks for computer vision, but their effectiveness depends heavily on how they are orchestrated: which tools are used, in what order, with what parameters, and under what visual conditions. Existing visual-programming agents typically generate a fixed solution pipeline, making them brittle under dense objects, occlusion, small targets, and domain shift. We introduce VTOS (Vision Tools Orchestration Search), a framework for adaptive visual tool orchestration through joint solution--observer search. VTOS co-searches executable solution programs that compose vision tools such as Grounding DINO, SAM, NMS, and slice-and-detect, together with observer programs that diagnose candidate solutions, identify failure modes, and generate actionable feedback. These observations are accumulated in a shared VisionThoughts knowledge base to guide subsequent search. We evaluate VTOS through two case studies: dense object counting on LVIS-Count and zero-shot plant-disease segmentation on PlantSeg-OOD, which stress different orchestration challenges including threshold calibration, NMS, slicing, mask refinement, and domain generalization. Across both tasks, VTOS outperforms static tool pipelines and agentic visual-programming baselines, showing that co-searching solutions and observers is an effective strategy for adapting vision tools to challenging computer vision tasks.

Problem

• vision tool orchestration
• adaptive visual programming
• domain shift
• dense object detection
• zero-shot segmentation

Innovation

• tool orchestration
• co-search
• vision foundation models
• adaptive visual programming
• observer feedback