VISUALSKILL: Multimodal Skills for Computer-Use Agents

2026-06-16

AI Summary

This work addresses the limitations of current agents in long-horizon tasks and unseen software environments, which primarily stem from skill libraries relying solely on textual descriptions while neglecting visual information from graphical user interfaces. To overcome this, the authors propose VISUALSKILL—a hierarchical, multimodal skill framework tailored to target applications—that uniquely preserves visual cues within skill artifacts. By integrating human-authored documentation with real-time UI exploration, VISUALSKILL constructs thematic,图文并茂 (text-and-image-rich) skills through a two-stage construction pipeline and supports on-demand loading via the load_topic MCP tool. Evaluated on the CUA-World and OSExpert-Eval benchmarks, the approach achieves an average score of 0.456, outperforming the no-skill baseline by 15.3 points and text-only skills by 8.3 points, substantially enhancing the agent’s ability to recognize UI elements and verify interface states.

Abstract

Computer-use agents (CUAs) approach human-level performance on standardised benchmarks but still struggle on long-horizon tasks and unseen software. Existing skill libraries address this with reusable skills, but represent the skill artifact as text only, despite the visual nature of GUI interaction. We propose VISUALSKILL: a hierarchical multimodal skill, tailored to each target application and organised as a central index over per-topic files, which the agent consumes through a load_topic MCP tool that fetches the relevant topic's text and figures on demand. We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration. On two CUA benchmarks, CUA-World and OSExpert-Eval, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL, a +15.3 point absolute lift over the no-skill baseline (0.303). Against a matched text-only skill that is generated from the same source content and differs from VISUALSKILL only in modality, VISUALSKILL yields a further +8.3 point absolute gain over the matched text-only skill (0.373 vs. 0.456), providing direct evidence that retaining visual figures in the skill artifact, rather than verbalizing them away, helps the agent both identify UI elements and verify workflow state after each action. Our code is available at https://github.com/XMHZZ2018/VisualSkills.

Problem

• computer-use agents
• GUI interaction
• multimodal skills
• long-horizon tasks
• skill representation

Innovation

• multimodal skills
• computer-use agents
• visual grounding
• GUI interaction
• skill library