Bidirectional Representations Augmented Autoregressive Biological Sequence Generation:Application in De Novo Peptide Sequencing

2025-10-09

AI Summary

Autoregressive (AR) models struggle to capture global bidirectional dependencies in biological sequences—e.g., de novo peptide sequencing and protein modeling—due to their unidirectional inductive bias, whereas non-autoregressive (NAR) models suffer from limited generation coherence and scalability. To address this trade-off, we propose BiAR, an enhanced autoregressive framework integrating bidirectional contextual representation. Its core innovation is a dual-decoder architecture: a shared encoder jointly serves a NAR decoder (for global context modeling) and an AR decoder (for stable, coherent sequence generation), coordinated via cross-decoder attention and importance-tempered training to enable synergistic optimization. Evaluated on a nine-species de novo peptide sequencing benchmark, BiAR substantially outperforms both pure AR and NAR baselines, achieving significant improvements in generalization capability and generation quality.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Autoregressive (AR) models, common in sequence generation, are limited in many biological tasks such as de novo peptide sequencing and protein modeling by their unidirectional nature, failing to capture crucial global bidirectional token dependencies. Non-Autoregressive (NAR) models offer holistic, bidirectional representations but face challenges with generative coherence and scalability. To transcend this, we propose a hybrid framework enhancing AR generation by dynamically integrating rich contextual information from non-autoregressive mechanisms. Our approach couples a shared input encoder with two decoders: a non-autoregressive one learning latent bidirectional biological features, and an AR decoder synthesizing the biological sequence by leveraging these bidirectional features. A novel cross-decoder attention module enables the AR decoder to iteratively query and integrate these bidirectional features, enriching its predictions. This synergy is cultivated via a tailored training strategy with importance annealing for balanced objectives and cross-decoder gradient blocking for stable, focused learning. Evaluations on a demanding nine-species benchmark of de novo peptide sequencing show that our model substantially surpasses AR and NAR baselines. It uniquely harmonizes AR stability with NAR contextual awareness, delivering robust, superior performance on diverse downstream data. This research advances biological sequence modeling techniques and contributes a novel architectural paradigm for augmenting AR models with enhanced bidirectional understanding for complex sequence generation. Code is available at https://github.com/BEAM-Labs/denovo.

Problem

• Overcoming unidirectional limitations in biological sequence generation models
• Integrating bidirectional context into autoregressive peptide sequencing frameworks
• Balancing generative coherence with global token dependency modeling

Innovation

• Hybrid framework combines AR and NAR decoders
• Cross-decoder attention integrates bidirectional contextual features
• Tailored training strategy ensures balanced and stable learning