Evolution Guided Generative Flow Networks

2024-02-03

arXiv.org

AI Summary

GFlowNets face significant training challenges in long-sequence generation and sparse-reward settings. To address this, we propose EA-GFlowNet—a novel framework that synergistically integrates gradient-free evolutionary algorithms (EAs) with GFlowNets for the first time. Specifically, EA-generated high-quality trajectories are used to construct a prioritized replay buffer, providing external guidance for flow-based policy learning. This mechanism substantially improves exploration efficiency under sparse rewards and enhances modeling capability for long paths. We evaluate EA-GFlowNet on diverse synthetic tasks and real-world benchmarks—including molecular generation and program synthesis—demonstrating marked improvements in sample quality and reward coverage. Notably, convergence accelerates by 2–5× on long-trajectory tasks. Our approach establishes a scalable, robust, gradient-free co-training paradigm for GFlowNets, advancing their applicability to complex structured output domains.

Technology Category

• Machine Learning: Evolutionary Learning
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generative Flow Networks (GFlowNets) are a family of probabilistic generative models that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks (EGFN), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA). Our method can work on top of any GFlowNets training objective, by training a set of agent parameters using EA, storing the resulting trajectories in the prioritized replay buffer, and training the GFlowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards.

Problem

• Enhance GFlowNets training for long time horizons
• Address sparse rewards in GFlowNets using Evolutionary algorithms
• Improve sampling of compositional objects with EA-augmented GFlowNets

Innovation

• Evolutionary algorithms enhance GFlowNets training
• Prioritized replay buffer stores EA trajectories
• Handles long trajectories and sparse rewards