Benchmarking and Evaluation of AI Models in Biology: Outcomes and Recommendations from the CZI Virtual Cells Workshop

2025-07-14

AI Summary

Biology lacks cross-domain, standardized AI model benchmarks, hindering model robustness and trustworthiness. To address this, we introduce the first multimodal AI benchmarking framework spanning imaging, transcriptomics, proteomics, and genomics—systematically tackling data heterogeneity, noise, bias, and resource fragmentation. Our approach integrates a high-fidelity data curation pipeline, unified preprocessing tools, biologically grounded multimodal evaluation metrics, and an open collaborative platform to enable fair, cross-task and cross-modal comparisons. A core innovation is the “virtual cell” paradigm—a biologically anchored, integrative evaluation framework—that unifies disparate modalities through shared cellular context. We further release a reproducible, extensible set of AI model evaluation guidelines. The framework significantly enhances rigor, transparency, and cross-domain comparability in biological AI research, accelerating AI-driven mechanistic discovery and therapeutic translation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Artificial intelligence holds immense promise for transforming biology, yet a lack of standardized, cross domain, benchmarks undermines our ability to build robust, trustworthy models. Here, we present insights from a recent workshop that convened machine learning and computational biology experts across imaging, transcriptomics, proteomics, and genomics to tackle this gap. We identify major technical and systemic bottlenecks such as data heterogeneity and noise, reproducibility challenges, biases, and the fragmented ecosystem of publicly available resources and propose a set of recommendations for building benchmarking frameworks that can efficiently compare ML models of biological systems across tasks and data modalities. By promoting high quality data curation, standardized tooling, comprehensive evaluation metrics, and open, collaborative platforms, we aim to accelerate the development of robust benchmarks for AI driven Virtual Cells. These benchmarks are crucial for ensuring rigor, reproducibility, and biological relevance, and will ultimately advance the field toward integrated models that drive new discoveries, therapeutic insights, and a deeper understanding of cellular systems.

Problem

• Lack of standardized benchmarks for AI models in biology
• Challenges in data heterogeneity, noise, and reproducibility
• Need for collaborative platforms to improve model evaluation

Innovation

• Standardized cross-domain benchmarking frameworks
• High-quality data curation and tooling
• Open collaborative platforms for evaluation