Random Policy Evaluation Uncovers Policies of Generative Flow Networks

2024-06-04

AI Summary

This work establishes a theoretical connection between Generative Flow Networks (GFlowNets) and standard reinforcement learning (RL), specifically through the lens of policy evaluation. Method: The authors first prove that, under zero-entropy regularization, the value function of a uniform random policy exactly recovers the GFlowNet flow function—thereby unifying GFlowNets with non-maximum-entropy RL. Leveraging this insight, they propose Random Policy Evaluation (RPE), a parameter-free algorithm that computes exact flow matching and reward matching solely via the value function of a fixed random policy. RPE integrates policy evaluation, flow matching, and generative modeling in a theoretically grounded and computationally lightweight framework. Contribution/Results: RPE achieves performance on par with state-of-the-art GFlowNet methods across multiple benchmark tasks, demonstrating that policy evaluation alone suffices for high-quality, diverse generative modeling—without explicit flow or policy optimization.

Technology Category

• Machine Learning: Reinforcement Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The Generative Flow Network (GFlowNet) is a probabilistic framework in which an agent learns a stochastic policy and flow functions to sample objects with probability proportional to an unnormalized reward function. GFlowNets share a strong connection with reinforcement learning (RL) that typically aims to maximize reward. A number of recent works explored connections between GFlowNets and maximum entropy (MaxEnt) RL, which incorporates entropy regularization into the standard RL objective. However, the relationship between GFlowNets and standard RL remains largely unexplored, despite the inherent similarities in their sequential decision-making nature. While GFlowNets can discover diverse solutions through specialized flow-matching objectives, connecting them to standard RL can simplify their implementation through well-established RL principles and also improve RL's capabilities in diverse solution discovery (a critical requirement in many real-world applications), and bridging this gap can further unlock the potential of both fields. In this paper, we bridge this gap by revealing a fundamental connection between GFlowNets and one of the most basic components of RL -- policy evaluation. Surprisingly, we find that the value function obtained from evaluating a uniform policy is closely associated with the flow functions in GFlowNets. Building upon these insights, we introduce a rectified random policy evaluation (RPE) algorithm, which achieves the same reward-matching effect as GFlowNets based on simply evaluating a fixed random policy, offering a new perspective. Empirical results across extensive benchmarks demonstrate that RPE achieves competitive results compared to previous approaches, shedding light on the previously overlooked connection between (non-MaxEnt) RL and GFlowNets.

Problem

• Explores connection between GFlowNets and standard RL.
• Introduces rectified random policy evaluation algorithm.
• Demonstrates competitive results in diverse solution discovery.

Innovation

• Generative Flow Network framework
• Rectified Random Policy Evaluation
• Connects GFlowNets and standard RL