The Agentic Garden of Forking Paths

2026-07-01

AI Summary

Scientific research often yields inconsistent conclusions due to the multiplicity of plausible analytical paths—a phenomenon known as “forking paths”—which are typically unobservable. This work proposes an AI-agent-based framework for exploring the analytical space: by endowing agents with distinct personality profiles and employing Agentic Bootstrap to sample diverse analytical trajectories, the approach introduces the multiverse value (m-value) to quantify the distribution of analytical outcomes. For the first time, this enables systematic evaluation and visualization of analytical path variability. In an immigration dataset experiment, the framework replicated 72% of the effect-size variation observed across human analysts with differing ideological leanings, and 13.5% of human-reported results fell within the most extreme 5% of the analytical distribution (m < 0.05), demonstrating the framework’s capacity to enhance the assessment of scientific credibility beyond traditional single-analysis paradigms.

Abstract

Empirical research rarely admits a unique analysis. Different analytical choices can lead to different conclusions from the same data, yet these hidden forking paths are difficult to observe. We show that AI agents capture much of the analytical variation among human researchers while making these paths explicit. Across four high-stakes domains, assigning different personas is sufficient for AI agents to report divergent, often opposing, conclusions from the same data and question, with findings systematically aligned with those beliefs. In a study in which 42 human research teams analyzed the same immigration dataset, AI agents reproduced 72% of the human ideological gap in reported effect estimates. Despite reaching opposing conclusions, it is difficult to identify clear issues in each analysis based on the final AI reports: 86% passed independent AI review and 78% passed majority human expert review. These findings suggest that the central challenge is often not flawed analyses, but selective exploration and reporting from a large space of methodologically defensible analyses. AI agents may amplify this longstanding problem by making such exploration inexpensive and scalable. To address this, we introduce the m-value (multiverse value), the probability that an analysis path would produce a claim at least as extreme as the reported one. We further introduce Agentic Bootstrap, which estimates the m-value by using AI agents to sample plausible analysis paths. Applied to the human immigration study, 13.5% of reported human analyses fell in the most extreme 5% of the analysis space (m<0.05). Scientific evidence should therefore be evaluated not only by a single reported analysis but also by its position within the distribution of analyses that could reasonably have been reported. Agentic Bootstrap makes this distribution observable and turns it into a criterion for scientific credibility.

Problem

• forking paths
• analytical variation
• selective reporting
• scientific credibility
• multiverse analysis

Innovation

• Agentic Bootstrap
• m-value
• analytical flexibility
• AI agents
• multiverse analysis