Content Hidden Behind Execution: Analyzing Public Scratch Projects at Runtime

2026-07-04

AI Summary

This study addresses the challenge that a substantial amount of sensitive content in public Scratch projects only becomes apparent at runtime, rendering static metadata insufficient for accurately assessing age-appropriateness and safety. To tackle this issue, the authors propose a novel runtime-aware, multidimensional annotation framework that decouples content category, risk level, evidence channel, revelation mechanism, and annotation confidence, thereby enabling dynamic evaluation of executable learning artifacts. Through a combination of manual auditing and stratified sampling—including keyword search, clustering-based exploration, and interactive testing—the team performed fine-grained annotations on 500 public projects. Results reveal that 93% of these projects required runtime exploration to uncover safety-relevant signals, with 77% relying on user interaction or hidden assets. This work provides empirical grounding and methodological support for content moderation, student engagement, and the construction of safety-focused datasets in educational contexts.

Abstract

Public Scratch projects are reused in computing education as classroom examples, remix sources, open-exploration materials, and research data. Curation often begins with titles, thumbnails, descriptions, tags, and remix links, but Scratch projects are executable learning artifacts. Content affecting age appropriateness can appear only after execution, gameplay progression, a failure state, user interaction, costume switching, audio playback, or a hidden event trigger. We study "runtime-revealed sensitive content" as a computing education curation challenge: educators and researchers need runtime evidence about what students may encounter when Scratch projects are used in these settings. We introduce a runtime-aware annotation scheme that separates content category, risk level, evidence channel, reveal mechanism, and annotation confidence. Using this scheme, we conducted an audit of 500 public Scratch projects sampled from curated candidates, taxonomy-guided keyword search, and follow-up exploration of project clusters surfaced during review. In this audit, 467 of 500 projects (93%) required runtime exploration beyond static metadata to surface the safety-relevant signal; 387 (77%) required interaction, gameplay progression, failure states, or hidden-asset and code inspection. As a targeted classroom and research curation audit, the study characterizes reveal mechanisms in a selected corpus rather than estimating platform-wide prevalence or making platform-level safety claims. The results show metadata-only screening leaves key evidence unresolved in executable youth media. By separating content type, severity, evidence location, and reveal pathway, this work supports classroom project selection, student exploration practices, dataset construction, and educator-facing screening tools for block-based programming communities.

Problem

• runtime-revealed content
• Scratch projects
• content curation
• sensitive content
• educational safety

Innovation

• runtime-revealed content
• annotation scheme
• Scratch projects
• sensitive content detection
• executable learning artifacts