Understanding Structural Representation in Foundation Models for Polymers

2025-12-08

AI Summary

Polymer foundation models are hindered by data scarcity and inconsistent molecular representations, with the core bottleneck being inadequate structural encoding capability. Method: We propose an enhanced SMILES-driven graph representation for polymers that precisely captures topological structure and connectivity, and introduce the first language-style pretraining architecture specifically designed for polymers. Contribution/Results: We empirically discover—first in the literature—that SMILES exhibits strong interpolation generalization in large chemical language models: even semantically invalid sequences yield high-accuracy predictions, challenging conventional chemical representation paradigms. Through SMILES-augmented graph encoding, systematic ablation studies, attention mechanism analysis, and error attribution, our model achieves state-of-the-art or near-state-of-the-art performance across 28 benchmark tasks. We further demonstrate that diverse SMILES variants exhibit high representational invariance, advancing polymer foundation models toward rational, interpretable representation design.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

From the relative scarcity of training data to the lack of standardized benchmarks, the development of foundation models for polymers face significant and multi-faceted challenges. At the core, many of these issues are tied directly to the structural representation of polymers and here, we present a new foundation model using a SMILES-based polymer graph representation. This approach allows representation of critical polymer architectural features and connectivity that are not available in other SMILES-based representations. The developed polymer foundation model exhibited excellent performance on 28 different benchmark datasets. Critical evaluation of the developed representation against other variations in control experiments reveals this approach to be a highly performant method of representing polymers in language-based foundation models. These control experiments also reveal a strong invariance of all SMILES representations, with many variations achieving state-of-the-art or near state-of-the-art performance, including those which are chemically or semantically invalid. Examination of error sources and attention maps for the evaluated representations corroborate the findings of the control experiments, showing that chemistry language models based on SMILES interpolate over all sequence space for prediction tasks, not only those of semantically valid inputs. Overall, this work highlights the importance of control experiments as a check on human-imposed assumptions that can limit rational design of both chemistry foundation models and their underlying structural representations.

Problem

• Develops a SMILES-based polymer graph representation for foundation models
• Addresses challenges in polymer structural representation and data scarcity
• Evaluates representation invariance and performance across benchmark datasets

Innovation

• SMILES-based polymer graph representation captures architectural features
• Model achieves high performance across 28 benchmark datasets
• Control experiments reveal invariance in SMILES representations for predictions