🤖 AI Summary
Current scientific protocol generation tasks (SPFT) lack automated evaluation frameworks and rely heavily on costly, time-intensive human assessment.
Method: We propose LLAM-EVAL—the first fully automated, zero-cost, plug-and-play evaluation method tailored for biology experimental protocol generation. It employs pseudocode alignment to model structured action constraints, enabling precise, grammar-aware protocol validation.
Contribution/Results: We introduce BIOPROT 2.0, a benchmark dataset supporting cross-model and cross-domain evaluation. Our framework achieves strong agreement with human judgments in biology (Pearson *r* = 0.92), empirically establishing GPT- and Cohere-based models as state-of-the-art for protocol generation. Furthermore, LLAM-EVAL demonstrates robust transferability to chemistry and materials science domains, confirming its generalizability beyond biology. The framework is publicly released to foster reproducible, scalable protocol evaluation across scientific disciplines.
📝 Abstract
Automated generation of scientific protocols executable by robots can significantly accelerate scientific research processes. Large Language Models (LLMs) excel at Scientific Protocol Formulation Tasks (SPFT), but the evaluation of their capabilities rely on human evaluation. Here, we propose a flexible, automatic framework to evaluate LLM's capability on SPFT: ProtocoLLM. This framework prompts the target model and GPT-4 to extract pseudocode from biology protocols using only predefined lab actions and evaluates the output of target model using LLAM-EVAL, the pseudocode generated by GPT-4 serving as a baseline and Llama-3 acting as the evaluator. Our adaptable prompt-based evaluation method, LLAM-EVAL, offers significant flexibility in terms of evaluation model, material, criteria, and is free of cost. We evaluate GPT variations, Llama, Mixtral, Gemma, Cohere, and Gemini. Overall, we find that GPT and Cohere is a powerful scientific protocol formulators. We also introduce BIOPROT 2.0, a dataset with biology protocols and corresponding pseudocodes, which can aid LLMs in formulation and evaluation of SPFT. Our work is extensible to assess LLMs on SPFT across various domains and other fields that require protocol generation for specific goals.