PlantBiMoE: A Bidirectional Foundation Model with SparseMoE for Plant Genomes

2025-12-07

AI Summary

Plant genomic modeling faces challenges of parameter redundancy and difficulty capturing bidirectional dependencies across DNA strands. This paper introduces PlantBiMoE, a lightweight and efficient foundational model that pioneers the integration of a bidirectional Mamba architecture with a sparse mixture-of-experts (SparseMoE) mechanism, enabling joint modeling of forward and reverse DNA strands while substantially reducing activated parameters. Trained and evaluated on the enhanced plant genomics benchmark MPGB, PlantBiMoE achieves state-of-the-art performance on 20 out of 31 downstream tasks, outperforming existing methods including AgroNT and PDLLMs in average performance. Key contributions are: (1) the first bidirectional state-space model specifically designed for plant genomes; (2) a structured sparsity mechanism that effectively balances representational capacity and computational efficiency; and (3) empirical validation of consistent performance gains from bidirectional modeling across diverse tasks—including functional element identification and variant effect prediction—demonstrating its broad applicability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Understanding the underlying linguistic rules of plant genomes remains a fundamental challenge in computational biology. Recent advances including AgroNT and PDLLMs have made notable progress although, they suffer from excessive parameter size and limited ability to model the bidirectional nature of DNA strands respectively. To address these limitations, we propose PlantBiMoE, a lightweight and expressive plant genome language model that integrates bidirectional Mamba and a Sparse Mixture-of-Experts (SparseMoE) framework. The bidirectional Mamba enables the model to effectively capture structural dependencies across both the forward and reverse DNA strands, while SparseMoE significantly reduces the number of active parameters, improving computational efficiency without sacrificing modeling capacity. We evaluated and tested our model on the Modified Plants Genome Benchmark (MPGB), an enhanced genomic benchmark, which consolidates 31 datasets across 11 representative tasks, with input sequence lengths ranging from 50 to 6,000 bp. Experimental results demonstrate that PlantBiMoE achieves the best performance on 20 out of 31 datasets and the average best when comparing with existing models. In summary, all above results demonstrate that our model can effectively represent plant genomic sequences, serving as a robust computational tool for diverse genomic tasks, while making substantive contributions to plant genomics, gene editing, and synthetic biology. The code is available at: https://github.com/HUST-Keep-Lin/PlantBiMoE

Problem

• Modeling bidirectional dependencies in plant DNA strands effectively
• Reducing excessive parameters in plant genome language models
• Improving computational efficiency for diverse genomic sequence tasks

Innovation

• Bidirectional Mamba captures DNA strand dependencies
• SparseMoE reduces parameters for computational efficiency
• Lightweight model excels on diverse genomic tasks