π€ AI Summary
This study systematically evaluates the practical benefits of pretraining in DNA language models for downstream genomic tasks and investigates the effectiveness of Byte Pair Encoding (BPE) tokenization. By comparing Transformer-based architectures (e.g., DNABERT2) with convolutional models (e.g., ConvNova) across a range of genomic fine-tuning benchmarks, the work provides the first empirical analysis of whether pretraining is necessary and how BPE compares to traditional k-mer representations. The findings indicate that the performance gains conferred by pretraining are limited and that BPE does not consistently outperform k-mer tokenization across all tasks. These results offer critical empirical insights for the design of foundational models in genomics, challenging prevailing assumptions about the universal advantages of large-scale pretraining and subword tokenization in this domain.
π Abstract
Recent breakthroughs in foundation models and Large Language Models (LLMs) have introduced new opportunities for studying and decoding genomic sequences. Several state-of-the-art approaches, such as DNABERT2, rely on transformer-based architectures, while others, such as ConvNova, still build upon more conventional convolutional models. However, systematic benchmark comparisons across these methods remain scarce. Given that transformer-based models require extensive and costly pretraining, it is crucial to evaluate whether their performance gains justify this overhead. Moreover, LLMs such as DNABERT2 typically rely on Byte Pair Encoding (BPE) tokenization, whose relevance for DNA sequence representation is still debated within the genomics community. In this work, we investigate three key questions: (i) do transformer-based models provide sufficient improvements on fine-tuning tasks upon heavy pretraining, (ii) what is the actual contribution of pretraining in this setting, and (iii) how does BPE tokenization impact performance on genomics-related tasks?