🤖 AI Summary
This work addresses the limited performance of existing GUI agents in drag-based interactions—such as text highlighting and slider manipulation—which stems primarily from the absence of large-scale, high-quality datasets capturing such operations. To bridge this gap, the authors present DragOn, the first systematically constructed benchmark dataset specifically designed for drag interactions, encompassing four representative scenarios with 286,000 training screenshots, 3.5 million task instances, and 2,000 evaluation samples. They further introduce an end-to-end drag localization training paradigm conditioned on screen images and natural language instructions. Fine-tuning the Qwen vision-language model on DragOn substantially improves performance on drag-related tasks, demonstrating the dataset’s effectiveness in enhancing model generalization to real-world digital interactions and filling a critical data void in complex, continuous GUI manipulation.
📝 Abstract
GUI agents - vision-based models that control desktops, web browsers, and mobile devices through graphical user interfaces - promise to automate a wide range of digital tasks. While million-scale datasets have enabled substantial progress on click-grounding, drag grounding (e.g. drag-and-drop, swipe, highlight) data remains an order of magnitude smaller and current models fall short on complex drag-based interactions. We introduce DragOn, a drag grounding benchmark and training dataset covering four domains: text highlighting, cell selection, element resizing and slider manipulation. The dataset comprises 286K training screenshots and 3.5M training tasks, plus a 2000-example held-out evaluation suite. We evaluate proprietary (GPT, Claude) and open-weight (Qwen, Kimi, Holo) models, as well as a Qwen VLM fine-tuned on our training data. Results suggest that our dataset could improve performance of state-of-the-art models on downstream computer-use tasks.