Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?

2026-05-15

AI Summary

Diffusion models face significant challenges in reinforcement learning (RL) fine-tuning, including high computational costs, weak preference alignment, and mismatched reward signals. This work proposes AdaScope, a plug-and-play method that adaptively selects when to apply RL intervention during the denoising process. It is the first to reveal substantial differences in the impact of various denoising stages on RL fine-tuning efficacy and dynamically identifies the optimal training interval based on structural evolution and semantic consistency, terminating training early upon performance saturation. Without modifying the model architecture, AdaScope reduces computational overhead by 59% while achieving a 66% improvement in generation performance over the current state-of-the-art, thereby simultaneously optimizing both efficiency and effectiveness.

Abstract

Despite strong image-generation performance, diffusion models' reconstruction objectives limit alignment with human preferences. RL enables such alignment through explicit rewards. However, most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action-reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, AdaScope adaptively identifies the optimal intervention timing for RL by perceiving the structural evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare 'dual benefit': a reduction in computational costs alongside a significant performance improvement. We offer theoretical grounds for the design of AdaScope. Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%.

Problem

• reinforcement learning
• diffusion models
• preference alignment
• denoising trajectory
• reward hacking

Innovation

• adaptive RL intervention
• diffusion model fine-tuning
• reward alignment
• computational efficiency
• denoising trajectory optimization