NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

2026-06-20

AI Summary

This work addresses the lack of effective benchmarks and evaluation methodologies for natural language to code (NL2Code) tasks in block-based, event-driven, and concurrent programming environments like Scratch. The authors introduce the first large-scale executable NL2Scratch benchmark, comprising over 310,000 real-world projects paired with semantically aligned natural language descriptions. They further propose a Slot-level Semantic Consistency (SAC) metric that moves beyond superficial token matching to enable fine-grained assessment of semantic correctness in generated programs. Experimental results reveal that even models achieving high F1 scores exhibit substantial deficiencies under SAC evaluation, particularly in critical slots involving actions, conditions, and numeric parameters—highlighting semantic errors that conventional metrics fail to capture.

Abstract

Block-based programming environments such as Scratch are widely used in early programming education, yet natural-language-to-code (NL2Code) research has focused primarily on text-based languages. Scratch programs are event-driven, visually compositional, and distributed across concurrent scripts, making conventional NL2Code assumptions and evaluation insufficient. We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation comprising 311,648 parser-valid NL--program pairs, whose program side is extracted from real Scratch projects and paired with semantically aligned NL descriptions. For reliable evaluation beyond surface overlap, we propose Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement between descriptions and programs. With SAC, we construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark. Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment: models achieving token-level F1 above 0.93 often fail to attain perfect SAC, particularly on longer examples. Errors concentrate on operational slots like actions, conditions, and numeric arguments, exposing failure modes largely invisible under conventional metrics.

Problem

• NL2Code
• block-based programming
• Scratch
• executable benchmark
• semantic evaluation

Innovation

• NL2Scratch
• block-based programming
• semantic alignment
• executable benchmark
• Scratch