ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions

2026-05-05

AI Summary

Existing causal reasoning benchmarks predominantly focus on local answers or graph structures, making it difficult to evaluate a model’s ability to generalize executable causal mechanisms from limited interventional data. This work proposes ReplaySCM—a novel benchmark comprising 1,300 tasks generated from implicit Boolean structural causal models (SCMs) in binary worlds—requiring systems to output executable mechanisms conforming to a constrained Boolean domain-specific language (DSL). Correctness is assessed not by syntactic form but by replay behavior under both training and held-out interventions. The benchmark introduces varied levels of structural disclosure and semantically equivalent yet behaviorally consistent alternative SCM tasks, emphasizing executable generalization over unique identifiability. Experiments reveal that state-of-the-art large language models perform adequately only when variable order is known, showing marked degradation under hidden orders or root structures; post hoc counterexample auditing shows all semantic alternatives fail consistency checks, with average predecessor pattern coverage rising to 1.0.

Abstract

Most causal benchmarks for language models score local answers or graph structure. We introduce ReplaySCM, a 1,300 item benchmark for executable causal mechanism induction from finite interventional evidence. Each item contains binary worlds generated by a latent fully observed acyclic Boolean structural causal model (SCM). A system must output a mechanism map in a restricted Boolean DSL; the submission is parsed, checked for legality and acyclicity, and replayed on training and held-out intervention worlds. Scoring uses replay behavior rather than formula strings, so syntactically different mechanisms receive credit when they behave correctly. ReplaySCM varies the structural information disclosed to the model through Ordered, Block-order, Hidden-order, and Hidden-roots settings, and includes Alternative-SCM tasks that supply a valid reference SCM and ask for a semantically distinct alternative that fits the training worlds, together with a separating intervention and witness. Frontier LLMs infer parts of the functional-parent structure, but held-out replay drops sharply when order or root structure is hidden. We also evaluate a matched support-audit ladder: Original, Extra Worlds, and Counterexample Audit (CEx), that raises mean local predecessor-pattern coverage from 0.8949 to 0.9815 to 1.0; under the audited searches, no discovered semantic alternative remains consistent with the training worlds. The Ordered/Hidden-order gap persists under this stronger evidence. ReplaySCM complements answer-level causal reasoning and graph-discovery benchmarks by evaluating executable replay generalization from finite interventional evidence, without claiming unique identification of the latent SCM.

Problem

• causal mechanism induction
• interventional evidence
• executable replay
• structural causal model
• benchmark

Innovation

• executable causal mechanism
• interventional evidence
• structural causal model
• behavioral equivalence
• causal generalization