BioNeMo Framework: a modular, high-performance library for AI model development in drug discovery

2024-11-15

arXiv.org

AI Summary

Training AI models for drug discovery—particularly protein language models (pLMs)—increasingly relies on large-scale GPU clusters, yet existing frameworks lack both efficiency and usability. Method: We introduce the first high-performance training framework tailored for biochemical AI, enabling scalable pLM development and deployment on hundred-GPU clusters. Built on PyTorch and Megatron-LM, it features a modular architecture supporting flexible integration of optimized data loading, distributed training, mixed-precision arithmetic, sequence parallelism, and high-throughput I/O. Contribution/Results: On 256 NVIDIA A100 GPUs, the framework completes pretraining of a 3-billion-parameter BERT-style pLM on over one trillion tokens in just 4.2 days—substantially lowering the barrier to large-scale pLM training. The framework is open-sourced, enhancing reproducibility and fostering collaborative innovation in computational biology.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Artificial Intelligence models encoding biology and chemistry are opening new routes to high-throughput and high-quality in-silico drug development. However, their training increasingly relies on computational scale, with recent protein language models (pLM) training on hundreds of graphical processing units (GPUs). We introduce the BioNeMo Framework to facilitate the training of computational biology and chemistry AI models across hundreds of GPUs. Its modular design allows the integration of individual components, such as data loaders, into existing workflows and is open to community contributions. We detail technical features of the BioNeMo Framework through use cases such as pLM pre-training and fine-tuning. On 256 NVIDIA A100s, BioNeMo Framework trains a three billion parameter BERT-based pLM on over one trillion tokens in 4.2 days. The BioNeMo Framework is open-source and free for everyone to use.

Problem

• Facilitates large-scale AI model training for drug discovery
• Integrates modular components into existing biology workflows
• Enables efficient pLM training on massive datasets

Innovation

• Modular high-performance library for AI drug discovery
• Facilitates large-scale GPU training for biology models
• Open-source framework with community-contributable components