MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets

2024-08-28

arXiv.org

AI Summary

This work addresses the insufficient exploration and propensity to get trapped in local modes exhibited by continuous-domain GFlowNets under black-box reward functions. We propose Adapted Metadynamics, a novel offline augmentation sampling strategy. It is the first to incorporate the principle of adaptive metadynamics into continuous GFlowNets, dynamically constructing a repulsive bias potential that penalizes previously visited regions and actively steers the flow network toward distant, high-reward modes尚未 covered. Unlike conventional offline strategies, it requires neither online interaction nor gradient information, and is fully compatible with arbitrary black-box rewards. On multiple continuous benchmark tasks, our method significantly accelerates convergence to the target distribution and robustly discovers distant high-reward modes missed by standard approaches, thereby improving sample diversity and state-space coverage.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Generative Flow Networks (GFlowNets) are a class of generative models that sample objects in proportion to a specified reward function through a learned policy. They can be trained either on-policy or off-policy, needing a balance between exploration and exploitation for fast convergence to a target distribution. While exploration strategies for discrete GFlowNets have been studied, exploration in the continuous case remains to be investigated, despite the potential for novel exploration algorithms due to the local connectedness of continuous domains. Here, we introduce Adapted Metadynamics, a variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains. We use Adapted Metadynamics as an exploration strategy for continuous GFlowNets. We show several continuous domains where the resulting algorithm, MetaGFN, accelerates convergence to the target distribution and discovers more distant reward modes than previous off-policy exploration strategies used for GFlowNets.

Problem

• Exploration strategies for continuous GFlowNets are understudied.
• Adapted Metadynamics enhances exploration in continuous domains.
• MetaGFN accelerates convergence and discovers distant reward modes.

Innovation

• Adapted Metadynamics for continuous GFlowNets
• Exploration strategy for distant reward modes
• Accelerates convergence in continuous domains