Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting

2025-03-06

AI Summary

Existing bioinformatics benchmarks inadequately assess large language models’ (LLMs) real-world cross-domain capabilities. Method: We introduce Bio-benchmark—the first prompt-driven, multi-domain evaluation suite covering 30 tasks across proteins, RNA, drugs, electronic health records, and traditional Chinese medicine—enabling native zero-shot and few-shot chain-of-thought evaluation. We propose a unified evaluation framework, BioFinder—a high-accuracy answer extraction tool improving accuracy by ~30%—and domain-specific, knowledge-aware prompt engineering strategies tailored for biological reasoning. Results: Comprehensive experiments systematically characterize the capability boundaries of state-of-the-art models including GPT-4o and Llama-3.1-70B, revealing critical weaknesses in biological reasoning. Bio-benchmark establishes an empirically grounded, reproducible assessment paradigm to guide the development of biology-specialized LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have become important tools in solving biological problems, offering improvements in accuracy and adaptability over conventional methods. Several benchmarks have been proposed to evaluate the performance of these LLMs. However, current benchmarks can hardly evaluate the performance of these models across diverse tasks effectively. In this paper, we introduce a comprehensive prompting-based benchmarking framework, termed Bio-benchmark, which includes 30 key bioinformatics tasks covering areas such as proteins, RNA, drugs, electronic health records, and traditional Chinese medicine. Using this benchmark, we evaluate six mainstream LLMs, including GPT-4o and Llama-3.1-70b, etc., using 0-shot and few-shot Chain-of-Thought (CoT) settings without fine-tuning to reveal their intrinsic capabilities. To improve the efficiency of our evaluations, we demonstrate BioFinder, a new tool for extracting answers from LLM responses, which increases extraction accuracy by round 30% compared to existing methods. Our benchmark results show the biological tasks suitable for current LLMs and identify specific areas requiring enhancement. Furthermore, we propose targeted prompt engineering strategies for optimizing LLM performance in these contexts. Based on these findings, we provide recommendations for the development of more robust LLMs tailored for various biological applications. This work offers a comprehensive evaluation framework and robust tools to support the application of LLMs in bioinformatics.

Problem

• Evaluate LLMs across diverse bioinformatics tasks effectively.
• Introduce Bio-benchmark for 30 key bioinformatics tasks.
• Propose strategies to optimize LLM performance in bioinformatics.

Innovation

• Developed Bio-benchmark for 30 bioinformatics tasks
• Introduced BioFinder for efficient answer extraction
• Proposed prompt engineering for LLM optimization