Representation Learning, Large-Scale 3D Molecular Pretraining, Molecular Property

2025-03-13

AI Summary

Existing molecular pre-trained representations (MPRs) predominantly rely on discrete atomic modeling, neglecting the continuous 3D spatial information occupied by molecules—thereby limiting their efficacy in few-shot drug discovery and materials design. To address this, we propose the first pre-training paradigm that explicitly models the complete 3D space surrounding a molecule, embodied in a novel Transformer architecture named SpaceFormer. Our method introduces adaptive grid-based spatial discretization, virtual point augmentation, efficient 3D positional encoding, and hierarchical grid sampling/merging mechanisms. Evaluated on multiple low-data molecular property prediction benchmarks, SpaceFormer consistently outperforms state-of-the-art 3D MPR methods, demonstrating both the effectiveness and generalizability of leveraging extranuclear spatial information. This work establishes a new direction for molecular representation learning by grounding structural understanding in the full 3D geometric context beyond atomic coordinates.

Technology Category

• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Molecular pretrained representations (MPR) has emerged as a powerful approach for addressing the challenge of limited supervised data in applications such as drug discovery and material design. While early MPR methods relied on 1D sequences and 2D graphs, recent advancements have incorporated 3D conformational information to capture rich atomic interactions. However, these prior models treat molecules merely as discrete atom sets, overlooking the space surrounding them. We argue from a physical perspective that only modeling these discrete points is insufficient. We first present a simple yet insightful observation: naively adding randomly sampled virtual points beyond atoms can surprisingly enhance MPR performance. In light of this, we propose a principled framework that incorporates the entire 3D space spanned by molecules. We implement the framework via a novel Transformer-based architecture, dubbed SpaceFormer, with three key components: (1) grid-based space discretization; (2) grid sampling/merging; and (3) efficient 3D positional encoding. Extensive experiments show that SpaceFormer significantly outperforms previous 3D MPR models across various downstream tasks with limited data, validating the benefit of leveraging the additional 3D space beyond atoms in MPR models.

Problem

• Addressing limited supervised data in drug discovery and material design.
• Incorporating 3D space beyond discrete atoms in molecular representations.
• Enhancing molecular pretrained representations using 3D space information.

Innovation

• Incorporates entire 3D molecular space
• Uses Transformer-based SpaceFormer architecture
• Implements grid-based space discretization