Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models

2026-04-19

AI Summary

Existing reward-based fine-tuning methods for diffusion and flow models lack a unified framework, leading to design complexity and hindered comparability. This work proposes Reward Score Matching (RSM), a principled framework that unifies diverse reward fine-tuning approaches as instances of score matching toward a reward-guided target distribution. By reformulating value-guided estimators and optimizing timestep selection strategies, RSM elucidates the inherent bias–variance–computation trade-offs and eliminates redundant mechanisms, thereby establishing a concise and interpretable design space. Empirical results demonstrate that RSM significantly improves alignment performance and computational efficiency under both differentiable and black-box reward settings, validating its generality and superiority.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reward-based fine-tuning aims to steer a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are motivated by different perspectives such as Soft RL, GFlowNets, etc., we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching toward a reward-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias--variance--compute tradeoffs of existing designs and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler redesigns that improve alignment effectiveness and compute efficiency across representative settings with differentiable and black-box rewards. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space.

Problem

• reward-based fine-tuning
• diffusion models
• flow models
• score matching
• generative modeling

Innovation

• Reward Score Matching
• Score Matching
• Reward-based Fine-tuning
• Diffusion Models
• Flow Models