NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction

2025-11-04

AI Summary

Current nucleotide foundation models face challenges in predicting functional fitness effects of sequence variants due to data heterogeneity and inconsistent preprocessing, hindering fair cross-family evaluation across DNA and RNA sequences. To address this, we introduce NABench—the first standardized benchmark for nucleic acid functional fitness prediction—systematically integrating 162 high-throughput experimental datasets (encompassing 2.6 million mutant sequences), with unified train/validation/test splits and comprehensive metadata annotation. NABench supports reproducible evaluation across zero-shot, few-shot, transfer learning, and fully supervised paradigms. We evaluate 29 representative models on NABench, revealing performance boundaries and cross-family transfer patterns between DNA- and RNA-centric tasks. The benchmark significantly improves assessment consistency, diversity, and reliability, establishing a robust baseline and comparable framework for synthetic biology and nucleic acid design.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Nucleotide sequence variation can induce significant shifts in functional fitness. Recent nucleotide foundation models promise to predict such fitness effects directly from sequence, yet heterogeneous datasets and inconsistent preprocessing make it difficult to compare methods fairly across DNA and RNA families. Here we introduce NABench, a large-scale, systematic benchmark for nucleic acid fitness prediction. NABench aggregates 162 high-throughput assays and curates 2.6 million mutated sequences spanning diverse DNA and RNA families, with standardized splits and rich metadata. We show that NABench surpasses prior nucleotide fitness benchmarks in scale, diversity, and data quality. Under a unified evaluation suite, we rigorously assess 29 representative foundation models across zero-shot, few-shot prediction, transfer learning, and supervised settings. The results quantify performance heterogeneity across tasks and nucleic-acid types, demonstrating clear strengths and failure modes for different modeling choices and establishing strong, reproducible baselines. We release NABench to advance nucleic acid modeling, supporting downstream applications in RNA/DNA design, synthetic biology, and biochemistry. Our code is available at https://github.com/mrzzmrzz/NABench.

Problem

• Standardizing nucleotide fitness prediction benchmarks for fair model comparison
• Evaluating foundation models across diverse DNA/RNA families and prediction scenarios
• Providing large-scale curated datasets to advance nucleic acid computational modeling

Innovation

• Large-scale benchmark for nucleic acid fitness prediction
• Standardized evaluation of 29 foundation models
• Aggregates 2.6 million sequences across DNA/RNA families