Relative Trajectory Balance is equivalent to Trust-PCL

2025-09-01

AI Summary

The theoretical relationship between Relative Trajectory Balance (RTB) and the off-policy KL-regularized reinforcement learning (RL) method Trust-PCL remains unclear, hindering unified understanding across generative modeling and regularized RL. Method: Through rigorous derivation of objective functions and formal equivalence proofs, we establish RTB’s precise position within the KL-regularized RL framework. We further validate our theoretical claims via analytical derivations and empirical fine-tuning experiments on diverse KL-regularized RL algorithms. Contribution/Results: We prove that RTB is fundamentally equivalent to Trust-PCL, naturally embedding it within both maximum-entropy RL and generative flow network optimization paradigms. Empirical results demonstrate that multiple KL-regularized RL algorithms—when applied to fine-tuning tasks—reproduce RTB’s performance, confirming the practical validity of the theoretical equivalence. This work provides a novel theoretical foundation for RL-based fine-tuning of generative models and bridges the conceptual gap between generative flow modeling and classical KL-regularized RL.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Recent progress in generative modeling has highlighted the importance of Reinforcement Learning (RL) for fine-tuning, with KL-regularized methods in particular proving to be highly effective for both autoregressive and diffusion models. Complementing this line of work, the Relative Trajectory Balance (RTB) objective was recently introduced in the context of Generative Flow Networks (GFlowNets) to serve the same role of improving fine-tuning in sequential generative models. Building on prior work linking GFlowNets and maximum-entropy RL, we establish in this paper an equivalence between RTB and Trust-PCL, an off-policy RL method with KL regularization. This equivalence situates RTB within the broader theoretical landscape of KL-regularized RL, and clarifies its relationship to earlier methods. Leveraging this insight, we revisit an illustrative example from the RTB paper and show that KL-regularized RL methods achieve comparable performance, offering an alternative perspective to what was previously reported.

Problem

• Establishes equivalence between Relative Trajectory Balance and Trust-PCL
• Links GFlowNet fine-tuning method to KL-regularized reinforcement learning
• Reevaluates performance claims through KL-regularized RL perspective

Innovation

• Establishes equivalence between RTB and Trust-PCL methods
• Leverages KL-regularized reinforcement learning for fine-tuning
• Applies off-policy RL to sequential generative models