Probing Chemical Language Models: Effects of Pre-training and Fine-tuning

2026-07-02

AI Summary

This study addresses the limited understanding of how chemical language models (CLMs) encode chemically meaningful molecular substructures during pretraining and fine-tuning. For the first time, it systematically evaluates the substructure awareness of eight pretrained and six randomly initialized CLMs across 78 distinct substructures, employing probing techniques to analyze how SMILES sequence representations evolve across model layers, complemented by downstream task fine-tuning experiments. The findings reveal that pretraining substantially enhances models’ comprehension of high-level molecular structures, while fine-tuning selectively strengthens representations of task-relevant substructures. Notably, even randomly initialized models effectively encode cyclic structures in their initial layers. This work elucidates the dynamic mechanisms underlying substructure representation in CLMs and offers novel insights into molecular representation learning.

Abstract

Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.

Problem

• Chemical Language Models
• Molecular Substructures
• Pre-training
• Fine-tuning
• SMILES

Innovation

• chemical language models
• molecular substructures
• pre-training
• fine-tuning
• representation probing