BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoning

2025-02-23

AI Summary

This work addresses the limited capability of large language models (LLMs) in dynamic biological pathway reasoning. We introduce BioMaze—the first benchmark dataset specifically designed for pathway reasoning—comprising 5.1K real-world research questions, and systematically identify LLMs’ shortcomings in perturbation response analysis and multi-scale causal inference. To overcome these limitations, we propose PathSeeker, a novel agent that pioneers an interactive, subgraph-guided progressive causal reasoning paradigm. PathSeeker integrates chain-of-thought prompting, graph-structure augmentation, and an interpretable subgraph retrieval mechanism. Experimental results demonstrate that PathSeeker achieves a 32.7% absolute accuracy improvement over strong baselines on pathway perturbation reasoning tasks, while significantly enhancing biological plausibility and interpretability. The BioMaze dataset, PathSeeker codebase, and trained models are fully open-sourced to foster reproducible research in computational biology and AI-driven pathway analysis.

Technology Category

• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Causal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The applications of large language models (LLMs) in various biological domains have been explored recently, but their reasoning ability in complex biological systems, such as pathways, remains underexplored, which is crucial for predicting biological phenomena, formulating hypotheses, and designing experiments. This work explores the potential of LLMs in pathway reasoning. We introduce BioMaze, a dataset with 5.1K complex pathway problems derived from real research, covering various biological contexts including natural dynamic changes, disturbances, additional intervention conditions, and multi-scale research targets. Our evaluation of methods such as CoT and graph-augmented reasoning, shows that LLMs struggle with pathway reasoning, especially in perturbed systems. To address this, we propose PathSeeker, an LLM agent that enhances reasoning through interactive subgraph-based navigation, enabling a more effective approach to handling the complexities of biological systems in a scientifically aligned manner. The dataset and code are available at https://github.com/zhao-ht/BioMaze.

Problem

• Enhancing LLMs for biological pathway reasoning
• Addressing LLM struggles in perturbed systems
• Developing interactive subgraph-based navigation for LLMs

Innovation

• LLMs for biological pathway reasoning
• Interactive subgraph-based navigation
• Dataset with complex pathway problems