OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking

2025-05-20

AI Summary

Genomic foundation models (GFMs) are advancing rapidly, yet face critical reproducibility bottlenecks—including opaque training data, non-interoperable model architectures, fragmented evaluation protocols, and insufficient interpretability. To address these challenges, we introduce the first modular benchmarking platform specifically designed for GFMs, unifying four core layers: data, models, evaluation, and interpretability. Our platform features standardized interface protocols, an automated evaluation pipeline, an open model integration framework, and an embedded interpretability toolkit. It enables one-click, end-to-end reproducible assessment across five major benchmark suites and has seamlessly integrated over 31 open-source GFMs. This work systematically resolves fundamental reproducibility barriers in genomic AI, significantly enhancing research credibility and cross-institutional collaboration efficiency.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Humans and AI: Other Foundations of Human Computation & AI
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The code of nature, embedded in DNA and RNA genomes since the origin of life, holds immense potential to impact both humans and ecosystems through genome modeling. Genomic Foundation Models (GFMs) have emerged as a transformative approach to decoding the genome. As GFMs scale up and reshape the landscape of AI-driven genomics, the field faces an urgent need for rigorous and reproducible evaluation. We present OmniGenBench, a modular benchmarking platform designed to unify the data, model, benchmarking, and interpretability layers across GFMs. OmniGenBench enables standardized, one-command evaluation of any GFM across five benchmark suites, with seamless integration of over 31 open-source models. Through automated pipelines and community-extensible features, the platform addresses critical reproducibility challenges, including data transparency, model interoperability, benchmark fragmentation, and black-box interpretability. OmniGenBench aims to serve as foundational infrastructure for reproducible genomic AI research, accelerating trustworthy discovery and collaborative innovation in the era of genome-scale modeling.

Problem

• Standardize evaluation of Genomic Foundation Models (GFMs)
• Address reproducibility challenges in genomic AI research
• Unify data, model, and benchmarking for GFMs

Innovation

• Modular platform for genomic model benchmarking
• Standardized one-command evaluation across benchmarks
• Automated pipelines for reproducibility and interoperability