Fair splits flip the leaderboard: CHANRG reveals limited generalization in RNA secondary-structure prediction

2026-03-20

AI Summary

Existing benchmarks for RNA secondary structure prediction struggle to faithfully evaluate models’ out-of-distribution generalization across diverse RNA families. To address this limitation, this work introduces CHANRG, a benchmark comprising 170,083 structurally non-redundant RNA sequences, featuring a novel fair partitioning strategy that integrates both structural and genomic information. It further establishes a multi-scale, symmetry-aware, padding-free evaluation framework. Systematic assessment of 29 methods reveals that while foundation models achieve state-of-the-art performance in-distribution, their out-of-distribution accuracy drops significantly; in contrast, structured decoders and direct neural predictors demonstrate greater robustness. This study thus provides a more reliable and rigorous standard for evaluating RNA secondary structure prediction methods.

Technology Category

• Machine Learning: Structured Learning
• Natural Language Processing: Safety and Robustness
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Accurate prediction of RNA secondary structure underpins transcriptome annotation, mechanistic analysis of non-coding RNAs, and RNA therapeutic design. Recent gains from deep learning and RNA foundation models are difficult to interpret because current benchmarks may overestimate generalization across RNA families. We present the Comprehensive Hierarchical Annotation of Non-coding RNA Groups (CHANRG), a benchmark of 170{,}083 structurally non-redundant RNAs curated from more than 10 million sequences in Rfam~15.0 using structure-aware deduplication, genome-aware split design and multiscale structural evaluation. Across 29 predictors, foundation-model methods achieved the highest held-out accuracy but lost most of that advantage out of distribution, whereas structured decoders and direct neural predictors remained markedly more robust. This gap persisted after controlling for sequence length and reflected both loss of structural coverage and incorrect higher-order wiring. Together, CHANRG and a padding-free, symmetry-aware evaluation stack provide a stricter and batch-invariant framework for developing RNA structure predictors with demonstrable out-of-distribution robustness.

Problem

• RNA secondary structure prediction
• generalization
• benchmark
• out-of-distribution robustness
• non-coding RNA

Innovation

• CHANRG
• RNA secondary structure prediction
• out-of-distribution robustness
• structure-aware deduplication
• foundation models