Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

2026-05-12

AI Summary

Neuroscience data are highly fragmented due to heterogeneous experimental paradigms and formats, severely impeding reuse. This study presents the first systematic evaluation of large language model–based AI agents in end-to-end reuse of real-world neural data. By integrating scientific papers, code, and datasets through prompt engineering, the agent automatically parses and reformats multi-source data to support neural–behavioral decoding tasks. The results show that while the agent can successfully execute individual subtasks, it struggles to complete the entire pipeline without errors. Moreover, its reliability as an evaluator is limited in the absence of ground truth. These findings highlight critical challenges for neural data sharing in the AI era and propose best practices centered on human–AI collaboration.

Abstract

Neuroscience data are highly fragmented across labs, formats, and experimental paradigms, and reuse often requires substantial manual effort. A persistent roadblock to data reuse and integration is the need to decipher bespoke and diverse data formatting choices. Common data formats have been proposed in response, but the field continues to struggle with a fundamental tension: formats flexible enough to accommodate diverse experiments are rarely descriptive enough to be self-explanatory, and sufficiently descriptive formats demand detailed documentation and curation effort that few labs can sustain. Agentic AI is a natural candidate to solve this problem: LLMs read code and text faster and with sustained attention to the low-level details humans tend to skim over. To measure how well agentic AI performs on this task, we selected eight recent papers studying large-scale mouse neural population recordings that shared both data and code, spanning diverse recording modalities, behavioral paradigms, and dataset formats (e.g., NWB, specialized APIs, and general-purpose Python or MATLAB files). We provided agents with the data, code, and paper, and prompted them to load, understand, and reformat the data for a common downstream task: training a decoder from neural activity to task or behavioral variables. General-purpose coding agents commonly used by scientists performed well on each sub-task, but rarely strung together a fully error-free end-to-end solution. We characterize the types of mistakes agents made and the dataset properties that elicited them, and propose data-sharing best practices for the agentic-AI era. We further find that agents-as-judges are unreliable at catching errors, especially without ground-truth references, so interactive, human-in-the-loop coding remains necessary.

Problem

• neuroscience data reuse
• data fragmentation
• common data formats
• agentic AI
• data integration

Innovation

• Agentic AI
• Neuroscience data reuse
• Data standardization
• LLM-based coding agents
• Human-in-the-loop