Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization

2026-07-01

AI Summary

This work addresses key challenges in molecular optimization, including unstable multi-step reasoning training, collapse during supervised fine-tuning, and sparse rewards in reinforcement learning. To overcome these issues, the authors propose an active inference paradigm that innovatively integrates active imitation learning with reinforcement learning, augmented by a dynamic reference update mechanism. This enables the policy to adaptively choose between imitation and exploration while continuously improving the quality of reference targets. Built upon the Group Relative Policy Optimization (GRPO) framework, the method alleviates performance bottlenecks imposed by fixed reference samples. Evaluated on the TOMG-Bench MOLOPT benchmark, the approach achieves a mean SRxSim of 0.1773, substantially outperforming GRPO (0.0959) and RePO (0.1665), and demonstrates statistically significant improvements across multiple metrics, including LogP, molecular weight (MR), and QED.

Abstract

Scientific reasoning is an increasingly important capability of large language models, yet improving the robustness and efficiency of training such reasoning remains a key open challenge. We study this problem in instruction-based molecular optimization, where answer-only supervised fine-tuning (SFT) collapses multi-step reasoning and reinforcement learning with verifiable rewards (RLVR) suffers from sparse feedback. Reference-guided Policy Optimization mitigates both by anchoring policy updates to dataset-provided references, but its effectiveness is tightly coupled to reference quality: weak or misaligned references impose a performance ceiling. To overcome this ceiling, we propose active reasoning, a paradigm in which the policy actively decides, on a per-instance basis, when to imitate a reference and when to reinforce its own discoveries, while continuously upgrading what it imitates. We instantiate this paradigm as Active Group Relative Policy Optimization (Active-GRPO), realized through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when the reference still outperforms the policy's own candidates, and shifts to self-improvement via reinforcement learning once the policy has generated molecules that surpass the reference. The latter continuously upgrades the reference itself by replacing it with the best policy-generated candidate discovered so far, progressively raising the imitation target and ensuring that reference guidance remains informative-rather than restrictive-throughout training. Across TOMG-Bench MOLOPT, Active-GRPO improves average SRxSim from 0.0959 for GRPO and 0.1665 for RePO to 0.1773 under matched three-seed evaluation, with statistically significant gains on LogP, MR, and QED.

Problem

• molecular optimization
• scientific reasoning
• reference-guided policy optimization
• reinforcement learning
• supervised fine-tuning

Innovation

• active reasoning
• molecular optimization
• imitation learning
• reinforcement learning
• reference-guided policy optimization