auto-psych: Automating the science of mind using agent-driven theory discovery and experimentation

2026-06-24

AI Summary

This study addresses the dual challenges of data collection bottlenecks and the lack of automation in theoretical discovery within psychological research by introducing the first end-to-end computational cognitive science framework. The proposed system employs a nested multi-agent architecture: an inner loop automatically constructs, fits, and critiques probabilistic cognitive models, while an outer loop autonomously designs and executes crowdsourced experiments to validate these models, thereby establishing a closed-loop “hypothesize–experiment–optimize” cycle. Integrating Bayesian model comparison with automated experimental design, the framework successfully recovers established theories on synthetic data and, through three rounds of human experimentation, discovers novel models that outperform existing ones in the literature. These results demonstrate a significant improvement in both the efficiency and accuracy of theoretical discovery, establishing the feasibility of automating scientific exploration in psychology.

Abstract

AI-based scientific automation is increasingly possible by using agents to generate hypotheses, design experiments, and analyze data. Data collection is a major bottleneck in this pipeline, however. Psychology, and computational cognitive science in particular, is well-positioned to benefit from AI experimentation because theories are often represented as code and crowdsourcing platforms enable programmatic human data collection at scale. Here, we apply automated discovery techniques to the project of generating theories in computational cognitive science, with an agent-based system collecting human data independently through crowdsourced survey experiments. As a testbed, we use a classic case study from cognitive psychology: judging which sequences of coin flips seem subjectively more random. Our system, auto-psych, uses nested agent-based discovery loops to generate explanatory theories of human behavior. The inner loop conjectures, fits, and critiques probabilistic cognitive models; the outer loop designs experiments to test these models, launches them online, and analyzes the data. This system can quickly and reliably recover ground-truth theories from synthetic data via systematic experimentation, but the nested structure is critical to model performance. Further, in three independent sequences of human experiments, the system finds theories that fit the data better than theories generated from the scientific literature. This work thus demonstrates the feasibility of automated data collection and theory discovery in computational cognitive science.

Problem

• scientific automation
• computational cognitive science
• theory discovery
• human data collection
• cognitive modeling

Innovation

• agent-based discovery
• automated theory generation
• computational cognitive science
• crowdsourced experimentation
• nested discovery loops