Unleashing Scientific Reasoning for Bio-experimental Protocol Generation via Structured Component-based Reward Mechanism

2025-10-17

AI Summary

Current large language models (LLMs) suffer from step omissions, logical inconsistencies, and semantic inaccuracies when generating biological experimental protocols, severely limiting their scientific utility. To address this, we propose a “sketch-then-fill” two-stage generation paradigm that decouples structural analysis, component construction, and natural language realization of protocols. We further design a fine-grained, structure-aware reward mechanism—grounded in atomic protocol units—to enable verifiable optimization of step alignment, sequential consistency, and semantic fidelity. Leveraging our newly curated large-scale biological protocol dataset, SciRecipe, we develop a knowledge-to-action staged training framework. Our model, Thoth, achieves state-of-the-art performance across multiple benchmark dimensions, significantly outperforming leading open-source and proprietary LLMs—particularly in logical step ordering and operational feasibility.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The foundation of reproducible science lies in protocols that are precise, logically ordered, and executable. The autonomous generation of these protocols through natural language queries could greatly improve the efficiency of the reproduction process. However, current leading large language models (LLMs) often generate incomplete or inconsistent protocols, limiting their utility. To address this limitation, we first introduce SciRecipe, a large-scale dataset of over 12K structured protocols spanning 27 biological subfields and encompassing both comprehension and problem-solving tasks. To further improve protocol generation, we propose the "Sketch-and-Fill" paradigm, which separates analysis, structuring, and expression to ensure each step is explicit and verifiable. Complementing this, the structured component-based reward mechanism evaluates step granularity, action order, and semantic fidelity, aligning model optimization with experimental reliability. Building on these components, we develop Thoth, trained through a staged Knowledge-to-Action process that progresses from knowledge acquisition to operational reasoning and ultimately to robust, executable protocol generation. Across multiple benchmarks, Thoth consistently surpasses both proprietary and open-source LLMs, achieving significant improvements in step alignment, logical sequencing, and semantic accuracy. Our approach paves the way for reliable scientific assistants that bridge knowledge with experimental execution. All data, code, and models will be released publicly.

Problem

• Generating precise biological protocols from natural language queries
• Addressing incomplete protocol generation by current language models
• Ensuring experimental reliability through structured reasoning mechanisms

Innovation

• Structured component-based reward mechanism for evaluation
• Sketch-and-Fill paradigm separates analysis and expression
• Staged Knowledge-to-Action process for protocol generation