Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

2026-06-18

AI Summary

This work proposes a method for automatically mining readable, structured skill libraries from user GUI interaction trajectories to enhance agent policy performance. The approach comprises a three-stage pipeline: trajectory segmentation, unsupervised clustering to generate candidate skills, and skill-aware policy training, integrating trajectory representation learning, offline reward modeling, and the GRPO algorithm. It presents the first systematic validation of the feasibility of unsupervised extraction of interpretable skills from real-world interactions. Experimental results on the InteraSkill Workflows benchmark show that five out of eight clusters achieve purity above 0.95, yet the resulting policy improvement remains marginal—skill-step accuracy in IW tasks increases only from 18.5% to 20.5%—highlighting a disconnect between skill interpretability and effective policy transfer.

Abstract

Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clusters are readable on the source benchmark: five of eight clusters have at least 0.95 purity against InteraSkill Workflows labels. However, readability does not imply transfer. GRPO improves IW skill-step accuracy only from 18.5\% to 20.5\%, leaves BrowseComp+ essentially unchanged, and underperforms trivial frequency priors on key source-domain metrics. We therefore present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.

Problem

• skill mining
• interaction trajectory
• computer-using agents
• policy transfer
• GUI automation

Innovation

• interaction trajectory mining
• skill library generation
• GUI automation
• skill-aware policy
• unsupervised skill clustering