Problem
Research questions and friction points this paper is trying to address.
long-context
RNA modeling
full-length RNAs
RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling
🤖 AI Summary
为解决现有模型无法处理长RNA序列的问题，本文提出了RIBOSPAN，一个具有10,240 nt上下文长度的预训练模型，通过结合双向自注意力机制和单核苷酸分词等技术，实现了全长RNA的高分辨率建模。
📝 Abstract
Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-transcript modeling at single-nucleotide resolution. We present RIBOSPAN, a 1.61-billion-parameter bidirectional RNA foundation model natively pretrained with context lengths up to 10,240 nt. RIBOSPAN combines dense bidirectional self-attention, single-nucleotide tokenization, and attention-isolated sequence packing to enable high-resolution modeling of complete long RNAs. We evaluate the model through nucleotide reconstruction, a controlled long-context representation benchmark, and frozen RNA-type representation analysis. Native 10K pretraining preserves strong reconstruction at 10,240 tokens, while continued pretraining with 40% masking improves recovery under heavy corruption while preserving representation quality. The long-context benchmark further shows that native 10K models maintain strong contextual responsiveness and context-specific representation separation while keeping perturbation-induced representation changes highly localized. Inference-time YaRN scaling recovers much of the contextual organization lost by direct extrapolation of short-context models, but induces substantially greater distal representation diffusion. Frozen-representation evaluations further demonstrate state-of-the-art RNA representation quality, with RIBOSPAN achieving the strongest overall performance across diverse RNA types and retaining a clear advantage on long RNAs. Building on the same backbone, we develop a multidimensionally conditioned discrete-diffusion framework for full-length mRNA generation and redesign, including synonymous-codon diffusion for protein-preserving CDS optimization. Together, RIBOSPAN establishes a powerful long-context foundation for transferable RNA representation learning and full-transcript mRNA design.
Innovation
Methods, ideas, or system contributions that make the work stand out.
long-context RNA modeling
bidirectional self-attention
single-nucleotide tokenization
full-length mRNA generation
representation learning
🔎 Similar Papers
No similar papers found.
