MMSkills: Towards Multimodal Skills for General Visual Agents

2026-05-13

AI Summary

Existing vision-based agents struggle to effectively model and reuse multimodal procedural knowledge, limiting their ability to reason about skills by integrating visual states, action sequences, and feedback in dynamic environments. This work proposes MMSkills, a framework that formalizes multimodal procedural knowledge for the first time. Leveraging trajectory clustering, procedural induction, and visual grounding, MMSkills automatically constructs compact skill packages—comprising textual procedures, state cards, and multi-view keyframes—from publicly available interaction data. A branch-loading mechanism enables lightweight, runtime multimodal alignment and retrieval. Experiments demonstrate that this approach significantly enhances the performance of both state-of-the-art and smaller multimodal models on GUI and gaming benchmarks, confirming that external procedural knowledge effectively augments the models’ internal priors.

Abstract

Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.

Problem

• multimodal procedural knowledge
• visual agents
• reusable skills
• state recognition
• visual evidence

Innovation

• multimodal procedural knowledge
• MMSkills
• visual agents
• skill generation
• state-conditioned skills