Automated Hypothesis Validation with Agentic Sequential Falsifications

2025-02-14

AI Summary

To address the challenge of manually verifying vast quantities of high-level abstract hypotheses generated by large language models (LLMs)—a process prone to hallucination and infeasible at scale—this paper proposes a multi-agent automated verification framework grounded in Popperian falsificationism. We introduce a novel sequential falsification testing paradigm that rigorously controls Type I error, integrating active evidence acquisition with cross-source observation (existing datasets plus targeted experiments) to enhance statistical power and scalability. Key innovations include: (1) an LLM-based collaborative agent architecture; (2) interpretable falsification experiment generation; (3) dynamic observational scheduling; and (4) a significance-driven sequential testing protocol. Empirical evaluation across six domains—biology, economics, sociology, and others—demonstrates that our framework achieves expert-level falsification performance while reducing verification time by one order of magnitude, all while maintaining robust error control and high statistical power.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, high-level statements that are difficult to validate directly. This challenge is further intensified by the rise of hypothesis generation from Large Language Models (LLMs), which are prone to hallucination and produce hypotheses in volumes that make manual validation impractical. Here we propose Popper, an agentic framework for rigorous automated validation of free-form hypotheses. Guided by Karl Popper's principle of falsification, Popper validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations, whether drawn from existing data or newly conducted procedures. We demonstrate Popper on six domains including biology, economics, and sociology. Popper delivers robust error control, high power, and scalability. Furthermore, compared to human scientists, Popper achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds, providing a scalable, rigorous solution for hypothesis validation.

Problem

• Automates validation of abstract hypotheses
• Controls Type-I errors in sequential testing
• Reduces validation time compared to humans

Innovation

• Agentic framework for hypothesis validation
• Sequential testing controls Type-I error
• LLM agents design falsification experiments