Follow the Mean: Reference-Guided Flow Matching

2026-05-11

AI Summary

Existing controllable generation methods often rely on fine-tuning, auxiliary networks, or test-time search, lacking a flexible, training-free control mechanism. This work proposes a “follow-the-mean” principle within the flow matching framework: by adjusting the conditional terminal mean and leveraging a reference set, it guides a pre-trained generative model to achieve desired attribute control. The approach employs deterministic interpolation-based flow matching, closed-form terminal mean correction, and semi-parametric guidance combining a frozen FLUX.2-klein model with a learnable residual refiner, enabling reference set swapping at inference time. Under fixed prompts, seeds, and weights, it effectively controls color, identity, style, and structure. Notably, this semi-parametric method attains unconditional DiT-B/4-level generation quality on AFHQv2.

Abstract

Existing approaches to controllable generation typically rely on fine-tuning, auxiliary networks, or test-time search. We show that flow matching admits a different control interface: adaptation through examples. For deterministic interpolants, the velocity field is solely governed by a conditional endpoint mean; shifting this mean shifts the flow itself. This yields a simple principle for controllable generation: steer a pretrained model by changing the reference set it follows. We instantiate this idea in two forms. Reference-Mean Guidance is training-free: it computes a closed-form endpoint-mean correction from a reference bank and applies it to a frozen FLUX.2-klein (4B) model, enabling control of color, identity, style, and structure while keeping the prompt, seed, and weights fixed. Semi-Parametric Guidance amortizes the same idea through an explicit mean anchor and learned residual refiner, matching unconditional DiT-B/4 quality on AFHQv2 while allowing the reference set to be swapped at inference time. These results point to a broader direction: generative models that adapt through data, not parameter updates.

Problem

• controllable generation
• flow matching
• reference-guided
• endpoint mean
• generative models

Innovation

• flow matching
• reference-guided generation
• endpoint mean
• training-free control
• semi-parametric guidance