🤖 AI Summary
This work addresses the lack of systematic evaluation benchmarks for AI agents in multi-step bioinformatics workflows, which hinders reliable assessment of their performance and robustness. We propose the first standardized evaluation framework encompassing end-to-end tasks such as RNA-seq analysis and variant calling. The framework integrates structured prompting, an automated LLM-based scoring mechanism, and perturbation tests—including corrupted inputs and decoy files—to holistically evaluate agents on workflow completeness, output correctness, and robustness, while also considering the applicability of open-source models in privacy-sensitive settings. Experimental results demonstrate that leading closed-source agents can reliably execute complex pipelines but exhibit reasoning vulnerabilities under perturbations, whereas open-source models, despite lower task completion rates, offer greater practical utility when data privacy constraints limit access to external systems.
📝 Abstract
This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) with prompts that specify concrete output artifacts to support automated assessment, including stress testing under controlled perturbations. We evaluate frontier closed-source and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that frontier agents can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, because bioinformatics workflows may involve sensitive patient data, proprietary references, or unpublished IP, closed-source models can be unsuitable under strict privacy constraints; in such settings, open-weight models may be preferable despite lower completion rates. We release the dataset and evaluation suite publicly.