Omni-DNA: A Unified Genomic Foundation Model for Cross-Modal and Multi-Task Learning

2025-02-05

AI Summary

Current genomic foundation models (GFMs) face two key bottlenecks: (1) task-specific fine-tuning incurs prohibitive computational overhead as model scale increases; and (2) rigid output formats limit generalization across diverse downstream tasks. To address these, we propose Omni-DNA—a unified genomic foundation model—introducing the first DNA autoregressive pretraining framework coupled with a multimodal task token expansion architecture. Omni-DNA supports cross-modal mapping (DNA→text/image) and joint learning across ten epigenomic tasks. Built upon Transformer architectures with scalable parameter counts (20M–1B), it achieves cross-modal semantic alignment via alignment-aware fine-tuning. On the Nucleotide Transformer and GB benchmarks spanning 26 tasks, Omni-DNA achieves state-of-the-art performance on 18. Multitask joint fine-tuning consistently outperforms single-task paradigms. Notably, Omni-DNA pioneers two novel cross-modal tasks: DNA2Function (generating functional descriptions from sequences) and Needle-in-DNA (sequence-to-image retrieval), both successfully demonstrated for the first time.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) demonstrate remarkable generalizability across diverse tasks, yet genomic foundation models (GFMs) still require separate finetuning for each downstream application, creating significant overhead as model sizes grow. Moreover, existing GFMs are constrained by rigid output formats, limiting their applicability to various genomic tasks. In this work, we revisit the transformer-based auto-regressive models and introduce Omni-DNA, a family of cross-modal multi-task models ranging from 20 million to 1 billion parameters. Our approach consists of two stages: (i) pretraining on DNA sequences with next token prediction objective, and (ii) expanding the multi-modal task-specific tokens and finetuning for multiple downstream tasks simultaneously. When evaluated on the Nucleotide Transformer and GB benchmarks, Omni-DNA achieves state-of-the-art performance on 18 out of 26 tasks. Through multi-task finetuning, Omni-DNA addresses 10 acetylation and methylation tasks at once, surpassing models trained on each task individually. Finally, we design two complex genomic tasks, DNA2Function and Needle-in-DNA, which map DNA sequences to textual functional descriptions and images, respectively, indicating Omni-DNA's cross-modal capabilities to broaden the scope of genomic applications. All the models are available through https://huggingface.co/collections/zehui127

Problem

• Unified genomic foundation model
• Cross-modal and multi-task learning
• Flexible output for diverse genomic tasks

Innovation

• Transformer-based auto-regressive genomic models
• Multi-task finetuning for diverse genomic applications
• Cross-modal capabilities mapping DNA to text and images