Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning

2026-05-28

AI Summary

Existing specialized vision models struggle to simultaneously achieve sophisticated language understanding and fine-grained perception of dense, small-scale objects, thereby hindering the advancement of general-purpose visual intelligence. This work proposes VisHarness—a trainable vision agent that decouples high-level reasoning from low-level execution and dynamically orchestrates heterogeneous expert models to accomplish multi-step visual tasks. The core innovation lies in a novel paradigm that prioritizes training the agent rather than the experts, augmented by a dynamic visual memory archival mechanism and lightweight online reinforcement learning for efficient policy optimization. Experimental results demonstrate that VisHarness significantly outperforms generalist models across four benchmark tasks and matches or exceeds the performance of specialized models.

Abstract

Recent progress in computer vision has produced a wide range of powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are usually optimized for isolated task formulations, making it difficult to directly support general-purpose visual intelligence, especially when a task requires complex language understanding and dense small-object perception. In this paper, we propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Instead of training a model to solve a specific visual task, VisHarness learns to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks. With only lightweight training, VisHarness learns a generalizable visual expert-harnessing policy and can solve common fundamental vision tasks under various complex conditions through multi-turn interactions with visual expert models. To enable efficient on-policy reinforcement learning training in a live environment, we introduce dynamic visual memory archiving, which mitigates the rapidly accumulating visual-token overhead caused by multi-turn interactions with visual expert models. Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared with task-specific models.

Problem

• multi-turn visual reasoning
• heterogeneous experts
• general-purpose visual intelligence
• dense small-object perception
• language-vision understanding

Innovation

• visual agent
• heterogeneous experts
• multi-turn reasoning
• dynamic visual memory
• expert harnessing