SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

2026-08-11

AI Summary

This work addresses the limitation of existing GUI agents in modeling visual procedural memory, which hinders their ability to recognize workflow states and critical interface elements. To overcome this, the authors propose Visual Skill Cards (VSCs)—a unified representation that encodes reusable interaction procedures along with their applicability cues, visual evidence, and validation signals. Building upon VSCs, they introduce two complementary frameworks: a retrieval-augmented executor for runtime inference and a retrieval-free CardDistill distillation approach for efficient policy transfer. Evaluated on Multimodal-Mind2Web and WebLINX-BrowserGym, the VSC-enhanced frozen GPT-4o-mini executor achieves a 11.6-point gain in Step Success Rate and a 2.9-point improvement in Overall Success Rate. Further, a distilled Qwen3-VL-2B student model via CardDistill yields additional gains of 12.0 and 3.2 points respectively, demonstrating the method’s effectiveness and generalizability.

Abstract

Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.

Problem

• visual procedural memory
• GUI action prediction
• interaction traces
• visual state
• skill representation

Innovation

• Visual Skill Cards
• Retrieval-Augmented Action Prediction
• On-Policy Distillation
• GUI Agent
• Multimodal Procedural Memory