Evaluating the Effectiveness of Cost-Efficient Large Language Models in Benchmark Biomedical Tasks

2025-07-18

AI Summary

The lack of systematic, cross-task evaluation frameworks hinders informed selection of large language models (LLMs) for biomedical applications. Method: We conduct a unified, standardized benchmarking study across 20+ open- and closed-source LLMs on biomedical text classification, generation, question answering, and multimodal image understanding tasks, measuring both performance and computational cost. Contribution/Results: We find no universally optimal model across tasks; lightweight open-source models—including BioMedLM and PubMedGPT—match or surpass proprietary models (e.g., GPT-4, Claude) on specific benchmarks while offering faster inference, flexible deployment, and enhanced data privacy. This is the first empirical demonstration of strong task dependency in biomedical LLM performance. We propose a “task-driven model selection” paradigm and release a reproducible evaluation benchmark with practical guidelines, enabling resource-efficient deployment in clinical decision support and biomedical research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

This paper presents a comprehensive evaluation of cost-efficient Large Language Models (LLMs) for diverse biomedical tasks spanning both text and image modalities. We evaluated a range of closed-source and open-source LLMs on tasks such as biomedical text classification and generation, question answering, and multimodal image processing. Our experimental findings indicate that there is no single LLM that can consistently outperform others across all tasks. Instead, different LLMs excel in different tasks. While some closed-source LLMs demonstrate strong performance on specific tasks, their open-source counterparts achieve comparable results (sometimes even better), with additional benefits like faster inference and enhanced privacy. Our experimental results offer valuable insights for selecting models that are optimally suited for specific biomedical applications.

Problem

• Evaluating cost-efficient LLMs in biomedical tasks
• Comparing closed-source and open-source LLMs' performance
• Identifying optimal LLMs for specific biomedical applications

Innovation

• Evaluates cost-efficient LLMs for biomedical tasks
• Compares closed-source and open-source LLMs performance
• Recommends optimal models for specific biomedical applications