TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology

2026-07-09

AI Summary

This work addresses the limitation of existing biological large language models, which suffer from a lack of unified, structured, high-quality pretraining corpora due to the heterogeneous and fragmented nature of biological data. To overcome this, we construct TheBioCollection, a standardized biological corpus comprising 52.6 billion tokens that systematically integrates multimodal data—including small molecules, proteins, genomes, cells, and pathways—and enriches it with computationally derived biological attributes via a bioinformatics toolchain. We further design novel instruction-tuning tasks targeting known model weaknesses and establish a corresponding evaluation benchmark. Using a fixed Gravity-16B-A3B architecture, our model achieves more than a twofold overall performance improvement on TheBioCollection-Eval, with significant gains across all subdomains while preserving its general language capabilities.

Abstract

The push toward large language models for biology (BioLM) has created a need for training corpora that can endow models with a genuine understanding of biology. However, existing biological resources, such as molecular databases, protein repositories, genomic annotations, single-cell atlases, and pathway databases, are scattered across heterogeneous formats and remain unorganized into a cohesive corpus for language model training. We present TheBioCollection, a 52.6B-token pre-training-scale corpus that converts these disparate resources into a unified, training-ready form spanning small molecules, proteins, genomic sequences, cells, and pathways. Beyond consolidating existing data, TheBioCollection enriches each record with tool-computed biological properties and introduces new instruction tasks for capabilities that current corpora barely cover. We pair the corpus with TheBioCollection-Eval, a matched suite probing recognition, generation, and prediction across molecular, protein, genomic, cellular, and cross-domain settings. Holding the base Gravity-16B-A3B architecture fixed, training on TheBioCollection more than doubles its overall score on TheBioCollection-Eval with gains in every domain, while leaving general linguistic ability nearly intact.

Problem

• biological resources
• heterogeneous formats
• unified corpus
• language model training
• BioLM

Innovation

• unified biological corpus
• large language models for biology
• instruction tuning
• multi-domain biological data integration
• pre-training scale