ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models

2025-05-18

AI Summary

Chemistry has long suffered from a scarcity of high-quality, large-scale, multimodal data, hindering the development of foundation models. To address this, we introduce ChemData—first large-scale, open multimodal dataset for chemistry foundation models (250 GB, 75B tokens), encompassing SMILES, SELFIES, IUPAC names, InChI strings, molecular images, pedagogical texts, scientific literature, executable code, and reasoning traces. We propose a novel, chemist-cognition-inspired hierarchical multimodal data architecture covering education-to-research stages; employ expert-driven curation, cross-source heterogeneous data alignment, structured metadata annotation, and license-compliant governance; and provide standardized evaluation splits and a unified Hugging Face API. ChemData significantly improves performance across molecular understanding, reaction prediction, and cross-modal reasoning, enabling end-to-end training and fair, reproducible benchmarking.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality datasets that reflect the field's multifaceted nature. We present the ChemPile, an open dataset containing over 75 billion tokens of curated chemical data, specifically built for training and evaluating general-purpose models in the chemical sciences. The dataset mirrors the human learning journey through chemistry -- from educational foundations to specialized expertise -- spanning multiple modalities and content types including structured data in diverse chemical representations (SMILES, SELFIES, IUPAC names, InChI, molecular renderings), scientific and educational text, executable code, and chemical images. ChemPile integrates foundational knowledge (textbooks, lecture notes), specialized expertise (scientific articles and language-interfaced data), visual understanding (molecular structures, diagrams), and advanced reasoning (problem-solving traces and code) -- mirroring how human chemists develop expertise through diverse learning materials and experiences. Constructed through hundreds of hours of expert curation, the ChemPile captures both foundational concepts and domain-specific complexity. We provide standardized training, validation, and test splits, enabling robust benchmarking. ChemPile is openly released via HuggingFace with a consistent API, permissive license, and detailed documentation. We hope the ChemPile will serve as a catalyst for chemical AI, enabling the development of the next generation of chemical foundation models.

Problem

• Lack of diverse, large-scale chemical datasets for foundation models
• Need for multimodal chemical data mirroring human learning progression
• Absence of standardized benchmarks for chemical AI model evaluation

Innovation

• Curated 250GB dataset for chemical foundation models
• Includes diverse chemical representations and modalities
• Openly released with standardized splits and API