ScratchWorld: Evaluating If World Models Compute Executable Consequences

2026-06-30

AI Summary

Existing evaluation methods for world models often misinterpret state copying as accurate prediction in environments with sparse changes, failing to assess a model’s understanding of executable causal relationships. To address this, this work proposes ScratchWorld—a novel offline diagnostic benchmark built upon the Scratch virtual machine—that enables structured replay of state transitions, latent variables, causal trajectories, and counterfactual outcomes through multimodal inputs and diverse diagnostic tasks. The study introduces a value-aware Changed-Field F1 metric that effectively distinguishes mere state replication from genuine causal reasoning. Evaluations on 659 samples reveal that even the best-performing among seven state-of-the-art models achieves only 13.8% Changed-Field F1, whereas a trivial copy strategy attains 98.0% full-state accuracy yet scores 0.0% on the Changed-Field F1, highlighting a fundamental deficiency in current models’ ability to adhere to executable rules and demonstrating the efficacy of the proposed evaluation framework.

Abstract

World-model evaluations often score a predicted future by overlap with a target state or observation. In sparse-change worlds, this can turn copied persistent state into apparent accuracy. We introduce ScratchWorld, an offline diagnostic benchmark that treats Scratch projects as executable worlds and uses a pinned Scratch VM to produce replay-verified transitions, hidden variables, causal traces, and counterfactual outcomes. ScratchWorld evaluates next-state prediction, long-horizon tracking, causal event attribution, and counterfactual prediction; each replay-verified target can be presented under raw-program, structured-state, natural-language, or rendered input modalities, and our experiments use the structured-state condition. Its primary state metric is value-aware changed-field $F_1$, which gives credit only for the changed field and its executed value. In a 659-example release, seven prompted language/reasoning models reach at most 13.8% value-aware changed-field $F_1$ in a state-only partial-observation stress test. A same-instance copy diagnostic makes the overlap confound concrete: copying the input state reaches 98.0% implied full-state field accuracy and 0.0% changed-field $F_1$, with the largest inflation on real projects. Auxiliary diagnostics separate hidden-state rollout drift, intervention sensitivity, causal attribution, and perturbation robustness. Across these settings, models often react to actions or interventions without following the executable rule that determines the changed value.

Problem

• world models
• executable consequences
• evaluation benchmark
• causal reasoning
• state prediction

Innovation

• world models
• executable consequences
• counterfactual prediction
• causal attribution
• changed-field F1