InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems

2026-03-16

AI Summary

This work addresses the absence of evaluation benchmarks for assessing large language models’ capacity to perform intervention reasoning and causal research design in real-world social systems. We introduce InterveneBench, the first end-to-end benchmark constructed from 744 empirical social science papers, which challenges models to infer policy intervention effects and articulate identification assumptions without access to predefined causal graphs. To enhance model performance on this task, we propose STRIDES, a multi-agent collaborative framework that substantially improves causal study design capabilities. Experimental results demonstrate that state-of-the-art large language models exhibit limited proficiency on InterveneBench, whereas STRIDES significantly outperforms existing approaches.

Technology Category

• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Causal Learning

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES.

Problem

• intervention reasoning
• causal inference
• large language models
• social science
• benchmarking

Innovation

• Intervention Reasoning
• Causal Inference
• Large Language Models
• Benchmark
• Multi-agent Framework