DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA

2024-12-06

Neural Information Processing Systems

AI Summary

Existing benchmarks inadequately assess the real-world capabilities of DNA language models (DNALMs) on critical regulatory DNA downstream tasks—including functional sequence discovery, cell-type-specific activity prediction, and counterfactual variant impact inference. Method: We introduce DART-Eval, the first comprehensive, regulation-focused evaluation benchmark for DNALMs, supporting zero-shot, probe-based, and fine-tuning paradigms. It integrates self-supervised DNALMs, *ab initio* baselines, and multi-level standardized tasks with biologically grounded metrics. Contribution/Results: Systematic evaluation reveals that state-of-the-art DNALMs show no significant performance advantage over lightweight baselines across most regulatory tasks, while incurring high computational cost and exhibiting unstable behavior. To address this, we propose a biology-informed evaluation framework that clarifies concrete directions for next-generation model development. All code and evaluation protocols are open-sourced to promote reproducibility and standardization of genomic AI assessment.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in self-supervised models for natural language, vision, and protein sequences have inspired the development of large genomic DNA language models (DNALMs). These models aim to learn generalizable representations of diverse DNA elements, potentially enabling various genomic prediction, interpretation and design tasks. Despite their potential, existing benchmarks do not adequately assess the capabilities of DNALMs on key downstream applications involving an important class of non-coding DNA elements critical for regulating gene activity. In this study, we introduce DART-Eval, a suite of representative benchmarks specifically focused on regulatory DNA to evaluate model performance across zero-shot, probed, and fine-tuned scenarios against contemporary ab initio models as baselines. Our benchmarks target biologically meaningful downstream tasks such as functional sequence feature discovery, predicting cell-type specific regulatory activity, and counterfactual prediction of the impacts of genetic variants. We find that current DNALMs exhibit inconsistent performance and do not offer compelling gains over alternative baseline models for most tasks, while requiring significantly more computational resources. We discuss potentially promising modeling, data curation, and evaluation strategies for the next generation of DNALMs. Our code is available at https://github.com/kundajelab/DART-Eval.

Problem

• Evaluating DNA language models on regulatory DNA tasks
• Assessing model performance in functional sequence discovery
• Comparing DNALMs with baseline models on computational efficiency

Innovation

• Introduces DART-Eval for regulatory DNA evaluation
• Assesses zero-shot, probed, fine-tuned scenarios
• Compares DNALMs against ab initio baselines