Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents

2026-06-29

AI Summary

Existing vision-language models rely on retraining or handcrafted prompts and tools for visual reasoning tasks, lacking a training-free adaptive mechanism. This work proposes Dynamo, a framework that, without updating model weights, dynamically evolves reusable cognitive skills and executable visual tools by analyzing successful and failed reasoning trajectories from a few annotated examples. Dynamo constructs a persistent capability library that enables automatic skill-tool pairing and continuous accumulation. It is the first approach to achieve training-free co-evolution of cognitive skills and perceptual tools in vision-language agents. Evaluated across four benchmarks and five models, Dynamo improves accuracy by 5.6% on average, outperforms all baselines in tool usage, closes 65–99% of the performance gap with reinforcement learning methods at minimal computational cost, and yields further gains when combined with RL.

Abstract

Improving vision-language models (VLMs) on visual reasoning typically requires retraining or hand-designed prompts and tools. We present Dynamo, a training-free framework that adapts a frozen VLM without any weight updates. On a small labeled training subset, the agent inspects its own correct and incorrect attempts and evolves two complementary capabilities: reusable reasoning skills for cognitive bottlenecks, and executable visual tools for perceptual ones. Each generated tool is paired with a skill that specifies when to invoke it, and both capability types accumulate in a persistent library. Across four visual reasoning benchmarks and five VLM backbones, Dynamo improves direct inference on all 20 model--benchmark settings (avg. +5.6 acc). When the tool set is given in advance, the framework learns when to call each tool, and per-step tool choice improves on every tested backbone. Against task-specific RL (VTool-R1, DeepEyes), Dynamo closes 65--99% of the RL gap at a fraction of the compute, and combines additively with RL when available.

Problem

• vision-language models
• visual reasoning
• tool use
• skill acquisition
• model adaptation

Innovation

• training-free adaptation
• dynamic skill-tool evolution
• vision-language agents
• visual reasoning
• executable visual tools