The Right Kind of Help: Evaluating the Effectiveness of Intervention Methods in Elementary-Level Visual Programming

2025-12-12

AI Summary

This study investigates the efficacy of three interventions—code-editing recommendations, editing-based quizzes, and metacognitive-strategy quizzes—in elementary visual programming learning, evaluated across both in-learning and post-learning phases. Employing a large-scale empirical design on the Hour of Code: Maze Challenge platform, multimodal data—including behavioral logs, pre/post-tests, and self-report scales—were collected and analyzed. For the first time, this work systematically disentangles and separately assesses intervention effects on immediate learning versus subsequent transfer performance. Results indicate that quiz-based interventions significantly enhance far-transfer outcomes without compromising problem-solving ability; all intervention groups outperformed the control group, with notable improvements in engagement and perceived skill growth. The core contribution lies in establishing the “post-learning phase” as a critical evaluation dimension and empirically validating the specificity of quiz-based interventions in promoting far transfer.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Prior work has explored various intervention methods for elementary programming. However, the relative impact of these methods during the learning and post-learning phases remains unclear. In this work, we present a large-scale study comparing the effectiveness of various intervention methods in elementary programming both during learning and on novel tasks post-learning. Specifically, we compare three intervention methods: code-edit recommendations (Code-Rec), quizzes based on code edits (Code-Quiz), and quizzes based on metacognitive strategies (Plan-Quiz), along with a no-intervention control (group None). A total of 398 students (across grades 4-7) participated in a two-phase study: learning phase comprising write-code tasks from the Hour of Code: Maze Challenge with the intervention, followed by a post-learning phase comprising more advanced write-code tasks without any intervention. All intervention methods significantly improved learning performance over the control group while preserving students' problem-solving skills in the post-learning phase. Quiz-based methods further improved performance on novel post-learning tasks. Students in intervention groups also reported greater engagement and perceived skill growth.

Problem

• Evaluates intervention methods in elementary visual programming
• Compares effectiveness during learning and post-learning phases
• Assesses impact on performance, problem-solving, and student engagement

Innovation

• Compares three intervention methods for programming
• Evaluates learning and post-learning phase effectiveness
• Uses large-scale study with elementary students