How evaluation choices distort the outcome of generative drug discovery

2024-12-24

Journal of Cheminformatics

AI Summary

Generative molecular design lacks standardized evaluation protocols, leading to unreliable benchmarking and inaccurate prospective screening. Method: Through systematic analysis of ~1 billion molecules, we identify library size as a critical confounding factor—common metrics (e.g., uniqueness, distributional similarity) exhibit strong size-dependent bias, yielding misleading conclusions. To address this, we propose computationally efficient, scale-invariant evaluation metrics; establish a robust framework for model comparison; provide practical guidelines for prospective molecular screening; and formally characterize the fundamental divergence between deep generative modeling objectives and drug discovery requirements under diversity constraints. Results: Empirical validation demonstrates that our new metrics substantially improve evaluation stability and reproducibility, offering a reliable, standardized assessment paradigm for generative drug discovery.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

“How to evaluate the de novo designs proposed by a generative model?” Despite the transformative potential of generative deep learning in drug discovery, this seemingly simple question has no clear answer. The absence of standardized guidelines challenges both the benchmarking of generative approaches and the selection of molecules for prospective studies. In this work, we take a fresh – critical and constructive – perspective on de novo design evaluation. By training chemical language models, we analyze approximately 1 billion molecule designs and discover principles consistent across different neural networks and datasets. We uncover a key confounder: the size of the generated molecular library significantly impacts evaluation outcomes, often leading to misleading model comparisons. We find increasing the number of designs as a remedy and propose new and compute-efficient metrics to compute at large-scale. We also identify critical pitfalls in commonly used metrics — such as uniqueness and distributional similarity — that can distort assessments of generative performance. To address these issues, we propose new and refined strategies for reliable model comparison and design evaluation. Furthermore, when examining molecule selection and sampling strategies, our findings reveal the constraints to diversify the generated libraries and draw new parallels and distinctions between deep learning and drug discovery. We anticipate our findings to help reshape evaluation pipelines in generative drug discovery, paving the way for more reliable and reproducible generative modeling approaches. Our work takes a step toward enhancing the robustness and reliability of evaluation practices in generative drug discovery. We systematically analyze current evaluation practices using approximately one billion designs from deep learning models. We find that the number of designs, often an overlooked parameter, can distort scientific outcomes related to distributional similarity and diversity. Moreover, we show that using larger design libraries than are typically adopted helps to avoid this pitfall, and we develop efficient algorithms to enable large-scale studies. We also propose guidelines for prospective molecule selection and uncover inherent constraints in diversifying molecular designs.

Problem

• Analyzing how evaluation metrics distort generative drug discovery outcomes
• Identifying library size as a key confounder in model comparisons
• Proposing refined strategies for reliable generative model evaluation

Innovation

• Proposed new compute-efficient metrics for large-scale evaluation
• Identified and addressed pitfalls in commonly used evaluation metrics
• Introduced refined strategies for reliable generative model comparison