RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

2026-06-28

AI Summary

This work addresses the limitation of existing agent skill repositories, which often rely on handcrafted rules or single-modality data and thus fail to fully leverage human multimodal knowledge. The authors propose a novel framework that systematically distills executable skills from diverse multimodal sources—including tutorial videos, code repositories, and textual articles—and organizes them into a hierarchical multimodal skill wiki to support agent retrieval, composition, and online expansion. By integrating multimodal understanding, skill distillation, and hierarchical knowledge organization, the approach achieves an average performance gain of 11.9 percentage points across seven creative domains. It significantly outperforms strong baselines in 26 out of 28 model–domain combinations, and ablation studies confirm the effectiveness of both multimodal inputs and the proposed architectural design.

Abstract

Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.

Problem

• executable skills
• multimodal resources
• software agents
• skill distillation
• tutorial videos

Innovation

• multimodal skill distillation
• executable agent skills
• Skill Wiki
• hierarchical organization
• online skill acquisition