URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

2026-07-06

AI Summary

Current retrosynthetic planning systems lack a flexible and chemically interpretable evaluation benchmark, making objective performance comparisons difficult. To address this gap, this work proposes the URSA evaluation framework, which introduces— for the first time—a dimension of chemical plausibility that simulates expert chemists’ judgment, complementing formal correctness. URSA enables a comprehensive assessment of retrosynthetic routes by considering reaction feasibility, pathway convergence, and other chemically relevant criteria. The framework evaluates both specialized models and large language models in realistic drug design scenarios, revealing that while the latter show promise in high-level strategic planning, they still significantly lag behind specialized models in task reliability. URSA thus establishes a new, interpretable, and holistic benchmark for the practical evaluation of retrosynthetic methodologies.

Abstract

Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Recent progress has produced both specialized deep-learning retrosynthesis systems and general-purpose large language models, but objective comparison remains difficult due to the lack of flexible, chemically interpretable benchmarking protocols. In the current study, we are introducing the URSA (Utilitarian RetroSynthesis Assessment) evaluation framework that provides the opportunity to benchmark the synthetic routes not only from a formal perspective, such as convergence to commercially available starting materials, but also from a chemical plausibility perspective, mimicking the way expert chemists evaluate the reactions and routes. The study covers a comprehensive evaluation of both conventional end-to-end retrosynthesis solutions and LLMs for the synthesis planning task on a set of novel, diverse target molecules with undisclosed synthetic routes, which represent realistic tasks in the daily drug design routine. We find that while LLMs can support high-level strategic planning, they currently underperform specialized retrosynthesis models in reliably solving synthesis planning tasks.

Problem

• retrosynthesis
• synthesis planning
• benchmarking
• drug discovery
• chemical plausibility

Innovation

• retrosynthesis benchmarking
• chemistry-aware evaluation
• utilitarian synthesis assessment
• large language models
• synthetic route plausibility