See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch

2026-02-11

AI Summary

This work addresses the lack of systematic evaluation of AI agents’ ability to construct programs through visual perception and interaction in graphical programming environments such as Scratch. The authors propose ScratchWorld, a benchmark grounded in the Use-Modify-Create pedagogical framework, comprising 83 tasks that assess multimodal GUI agents across four dimensions: creation, debugging, extension, and computation. A novel execution-driven runtime verification mechanism and dual interaction modes—primitive and composite—are introduced to effectively decouple program planning from GUI manipulation. Experimental results demonstrate that while state-of-the-art agents exhibit strong planning capabilities, they struggle significantly with fine-grained GUI operations, thereby highlighting the challenge and efficacy of the proposed benchmark.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Multiagent Systems: Modeling other Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Block-based programming environments such as Scratch play a central role in low-code education, yet evaluating the capabilities of AI agents to construct programs through Graphical User Interfaces (GUIs) remains underexplored. We introduce ScratchWorld, a benchmark for evaluating multimodal GUI agents on program-by-construction tasks in Scratch. Grounded in the Use-Modify-Create pedagogical framework, ScratchWorld comprises 83 curated tasks spanning four distinct problem categories: Create, Debug, Extend, and Compute. To rigorously diagnose the source of agent failures, the benchmark employs two complementary interaction modes: primitive mode requires fine-grained drag-and-drop manipulation to directly assess visuomotor control, while composite mode uses high-level semantic APIs to disentangle program reasoning from GUI execution. To ensure reliable assessment, we propose an execution-based evaluation protocol that validates the functional correctness of the constructed Scratch programs through runtime tests within the browser environment. Extensive experiments across state-of-the-art multimodal language models and GUI agents reveal a substantial reasoning--acting gap, highlighting persistent challenges in fine-grained GUI manipulation despite strong planning capabilities.

Problem

• multimodal GUI agents
• block-based programming
• program-by-construction
• Scratch
• evaluation benchmark

Innovation

• multimodal GUI agents
• ScratchWorld
• program-by-construction
• execution-based evaluation
• visuomotor control