Language Models Can Understand Spectra: A Multimodal Model for Molecular Structure Elucidation

2025-08-04

AI Summary

Existing molecular structure elucidation methods heavily rely on prior spectral databases and expert knowledge, limiting their ability to handle novel or unknown compounds; moreover, multi-spectroscopic (e.g., NMR, MS, IR) integration remains costly and poorly automated. To address these challenges, we propose SpectraLLM—the first large language model (LLM) designed for multimodal spectroscopic joint reasoning and the first to integrate LLMs into molecular spectral analysis. SpectraLLM constructs a unified continuous–discrete spectroscopic semantic space, enabling cross-modal substructure pattern recognition and end-to-end molecular structure generation. Its encoder–LLM collaborative architecture supports both single- and multi-spectrum inputs. Evaluated on six public chemical datasets, SpectraLLM consistently outperforms unimodal baselines in accuracy, generalization, and robustness—while substantially reducing dependence on domain expertise.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Structure elucidation is a fundamental technique for understanding the microscopic composition of matter and is widely applied across various disciplines in the natural sciences and engineering. However, existing methods often rely heavily on prior databases or known structural information, making it difficult to resolve unknown structures. In addition, complex structures typically require the joint analysis of multiple spectroscopic modalities. This process heavily depends on expert domain knowledge and is often accompanied by high costs in terms of both time and instrumentation. To address these challenges, we propose SpectraLLM, the first large language model designed to support multi-modal spectroscopic joint reasoning. SpectraLLM is capable of processing either single or multiple spectroscopic inputs and performing end-to-end structure elucidation. By integrating continuous and discrete spectroscopic modalities into a shared semantic space, SpectraLLM learns to uncover substructural patterns that are consistent and complementary across spectra, enabling precise molecular structure elucidation. We pretrain and fine-tune SpectraLLM in the domain of small molecules, and evaluate it on six standardized, publicly available chemical datasets. The model achieves state-of-the-art performance, significantly outperforming existing approaches trained on single modalities. Notably, SpectraLLM demonstrates strong robustness and generalization even for single-spectrum inference, while its multi-modal reasoning capability further improves the accuracy of structural prediction.

Problem

• Resolving unknown molecular structures without prior databases
• Joint analysis of multiple spectroscopic modalities efficiently
• Reducing reliance on expert knowledge and high costs

Innovation

• Multimodal LLM for spectroscopic joint reasoning
• Integrates spectra into shared semantic space
• End-to-end molecular structure elucidation