Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition

2026-07-01

AI Summary

Existing deterministic few-step flow-mapping generators struggle to directly adopt likelihood-ratio–based reinforcement learning (RL) post-training methods due to the absence of stochastic trajectories. To address this limitation, this work proposes the Flow-Map GRPO framework, which introduces an Anchored Stochastic Flow-Mapping Composition (ASFMC) mechanism. ASFMC injects conditional resampling stochasticity while preserving the original marginal probability paths, enabling online RL alignment of native deterministic few-step generators without modifying model architecture or requiring retraining. The method is compatible with various few-step flow models—including MeanFlow and sCM—as well as the FLUX text-to-image architecture, consistently achieving significant improvements over pretrained models in reward alignment, perceptual quality, and task-specific metrics.

Abstract

Few-step flow-map generators, such as consistency models and MeanFlow, accelerate sampling by directly learning long-range transport maps between noise and data. However, these models are typically deterministic, which makes them difficult to optimize with reinforcement learning (RL) post-training methods that require stochastic trajectories and well-defined likelihood ratios. Existing SDE-based stochasticization techniques are designed for velocity-based samplers with infinitesimal or finely discretized transitions, and therefore do not directly apply to long-range flow maps. In this work, we propose Flow-Map GRPO, an online RL post-training framework for deterministic few-step flow-map generators. The key component is Anchored Stochastic Flow Map Composition (ASFMC), a path-preserving stochasticization mechanism that introduces randomness through anchor-based conditional resampling while preserving the original marginal probability path of the deterministic flow map. We derive GRPO objectives for both single-time and two-time flow-map parameterizations. Experiments on few-step FLUX-based text-to-image generators, including MeanFlow and sCM, show that Flow-Map GRPO improves pretrained deterministic flow-map models across reward-based, perceptual, and task-level evaluation metrics. Our results demonstrate that deterministic few-step flow-map generators can be effectively aligned with RL post-training without modifying their original model parameterization or retraining them as native stochastic models.

Problem

• few-step flow-map generators
• reinforcement learning
• stochasticization
• deterministic models
• likelihood ratios

Innovation

• Flow-Map GRPO
• Anchored Stochastic Flow Map Composition
• reinforcement learning post-training
• few-step flow-map generators
• stochasticization