Raven: Rethinking Automated Assessment for Scratch Programs via Video-Grounded Evaluation

2026-04-20

AI Summary

This study addresses the challenge of automatically assessing event-driven, visual block-based programming assignments—such as those created in Scratch—which exhibit high implementation diversity and resist conventional evaluation approaches. Existing methods relying on assertions or predefined test cases suffer from limited scalability and insufficient robustness. To overcome these limitations, this work proposes a task-level, rule-driven automated assessment framework that innovatively integrates large language models with video analysis techniques. By observing the visual and interactive behaviors exhibited during program execution, the framework enables behavior-level scoring without requiring pre-specified test cases or direct inspection of code states or outputs. Evaluated on 13 real-world assignments comprising over 140 student submissions, the approach significantly outperforms existing tools in both accuracy and robustness. Classroom studies involving 30 students and 10 instructors further confirm its high usability and practical value.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Evaluation and Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Block-based programming environments such as Scratch are widely used in introductory computing education, yet scalable and reliable automated assessment remains elusive. Scratch programs are highly heterogeneous, event-driven, and visually grounded, which makes traditional assertion-based or test-based grading brittle and difficult to scale. As a result, assessment in real Scratch classrooms still relies heavily on manual inspection and delayed feedback, introducing inconsistency across instructors and limiting scalability. We present Raven, an automated assessment framework for Scratch that replaces program-specific state assertions with instructor-specified, task-level video generation rules shared across all student submissions. Raven integrates large language models with video analysis to evaluate whether a program's observed visual and interactive behaviors satisfy grading criteria, without requiring explicit test cases or predefined outputs. This design enables consistent evaluation despite substantial diversity in implementation strategies and interaction sequences. We evaluate Raven on 13 real Scratch assignments comprising over 140 student submissions with ground-truth labels from human graders. The results show that Raven significantly outperforms prior automated assessment tools in both grading accuracy and robustness across diverse programming styles. A classroom study with 30 students and 10 instructors further demonstrates strong user acceptance and practical applicability. Together, these findings highlight the effectiveness of task-level behavioral abstractions for scalable assessment of open-ended, event-driven programs.

Problem

• automated assessment
• Scratch programs
• event-driven programming
• scalable evaluation
• visual grounding

Innovation

• automated assessment
• video-grounded evaluation
• block-based programming
• large language models
• behavioral abstraction