A Reinforcement Learning-Driven Transformer GAN for Molecular Generation

2025-03-17

AI Summary

To address the sensitivity of SMILES representations and training instability in discrete generative adversarial networks (GANs) for molecular generation, this paper proposes RL-MolGAN: a decoder-first, Transformer-based discrete GAN framework supporting both *de novo* and scaffold-guided molecular design. We introduce the first deep integration of proximal policy optimization (PPO) and Monte Carlo tree search (MCTS) into discrete GAN training, and further develop RL-MolWGAN—a variant incorporating the Wasserstein distance and mini-batch discrimination to enhance convergence stability. Evaluated on QM9 and ZINC, RL-MolGAN achieves 98.7% syntactic validity for generated molecules, significantly outperforming baselines in uniqueness and quantitative estimate of drug-likeness (QED). Molecular diversity improves by 32%, and optimization of key physicochemical properties converges 2.1× faster.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Generating molecules with desired chemical properties presents a critical challenge in fields such as chemical synthesis and drug discovery. Recent advancements in artificial intelligence (AI) and deep learning have significantly contributed to data-driven molecular generation. However, challenges persist due to the inherent sensitivity of simplified molecular input line entry system (SMILES) representations and the difficulties in applying generative adversarial networks (GANs) to discrete data. This study introduces RL-MolGAN, a novel Transformer-based discrete GAN framework designed to address these challenges. Unlike traditional Transformer architectures, RL-MolGAN utilizes a first-decoder-then-encoder structure, facilitating the generation of drug-like molecules from both $de~novo$ and scaffold-based designs. In addition, RL-MolGAN integrates reinforcement learning (RL) and Monte Carlo tree search (MCTS) techniques to enhance the stability of GAN training and optimize the chemical properties of the generated molecules. To further improve the model's performance, RL-MolWGAN, an extension of RL-MolGAN, incorporates Wasserstein distance and mini-batch discrimination, which together enhance the stability of the GAN. Experimental results on two widely used molecular datasets, QM9 and ZINC, validate the effectiveness of our models in generating high-quality molecular structures with diverse and desirable chemical properties.

Problem

• Generating molecules with desired chemical properties.
• Overcoming challenges in applying GANs to discrete molecular data.
• Enhancing stability and optimizing properties in molecular generation.

Innovation

• Transformer-based GAN for molecular generation
• Reinforcement learning enhances GAN stability
• Wasserstein distance improves model performance