MolLangBench: A Comprehensive Benchmark for Language-Prompted Molecular Structure Recognition, Editing, and Generation

2025-05-21

AI Summary

Current AI models exhibit weak performance in language-driven molecular structure recognition, editing, and generation—particularly in generation, where accuracy is only 29.0%—revealing fundamental limitations in molecular understanding. Method: We introduce MolLangBench, the first comprehensive benchmark for the language–molecule interface, covering three modalities: SMILES strings, molecular images, and graph representations. It emphasizes high-quality, unambiguous, and verifiable evaluation. We propose a unified three-task framework (recognition, editing, generation) and ensure data rigor via automated sample construction using RDKit and iterative validation by domain experts. Contribution/Results: Experiments show that state-of-the-art models achieve only 79.2% and 78.5% accuracy on recognition and editing tasks, respectively—significantly below human performance—highlighting critical model deficiencies. MolLangBench provides a reliable, standardized evaluation protocol and concrete directions for advancing chemically grounded AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Precise recognition, editing, and generation of molecules are essential prerequisites for both chemists and AI systems tackling various chemical tasks. We present MolLangBench, a comprehensive benchmark designed to evaluate fundamental molecule-language interface tasks: language-prompted molecular structure recognition, editing, and generation. To ensure high-quality, unambiguous, and deterministic outputs, we construct the recognition tasks using automated cheminformatics tools, and curate editing and generation tasks through rigorous expert annotation and validation. MolLangBench supports the evaluation of models that interface language with different molecular representations, including linear strings, molecular images, and molecular graphs. Evaluations of state-of-the-art models reveal significant limitations: the strongest model (o3) achieves $79.2%$ and $78.5%$ accuracy on recognition and editing tasks, which are intuitively simple for humans, and performs even worse on the generation task, reaching only $29.0%$ accuracy. These results highlight the shortcomings of current AI systems in handling even preliminary molecular recognition and manipulation tasks. We hope MolLangBench will catalyze further research toward more effective and reliable AI systems for chemical applications.

Problem

• Evaluate molecule-language interface tasks: recognition, editing, generation
• Assess AI models handling molecular representations like strings, images, graphs
• Highlight limitations of current AI in molecular manipulation tasks

Innovation

• Automated cheminformatics tools for recognition tasks
• Expert-curated editing and generation tasks
• Supports multiple molecular representation interfaces