CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

2026-05-25

AI Summary

This study investigates whether large language model agents genuinely understand causal mechanisms or merely rely on predictive capabilities. To this end, the authors develop a scalable, interactive environment for causal discovery that requires agents to infer structural causal models from both observational and interventional data and generalize to novel scenarios. They propose a novel evaluation framework that decouples predictive accuracy from the ability to recover true causal graphs and introduce a domain-specific language to trace the evolution of agents’ causal hypotheses. Experimental results reveal that even the strongest current models (e.g., GPT-5.2-high) achieve 92% prediction accuracy yet attain only a 0.471 F1 score in causal graph recovery. Combining observational and interventional strategies simultaneously improves both metrics to approximately 80%, and maintaining hypothesis consistency effectively mitigates premature convergence in reasoning.

Abstract

We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is supported by a correct hypothesis about the underlying causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. CausaLab also includes a domain-specific language that records the agent's evolving SCM hypothesis, making trajectories inspectable and comparable with ground truth. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge $F_1$. This observation further motivates our exploration of different interaction strategies: Mixed observation--intervention strategies improve structural fidelity: in the mixed 6-node setting, GPT-5.2-high achieves 80% on both task accuracy and all-edge $F_1$. Yet even strong agents struggle to design informative interventions, as pure intervention strategies perform poorly on both task accuracy and all-edge $F_1$. We identify premature stopping as a major weakness of agents, and show that asking the model to verify the consistency between its hypothesis and past data can help mitigate this issue. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.

Problem

• causal discovery
• LLM agents
• structural causal model
• interactive reasoning
• causal understanding

Innovation

• interactive causal discovery
• structural causal model
• LLM agents
• causal reasoning
• intervention strategies