FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation

2026-05-19

AI Summary

This work addresses the challenge of enabling efficient bidirectional vision–language generation in existing text-to-image models without compromising their strong image priors. The authors propose FullFlow, which extends a pretrained Rectified Flow text-to-image model into a bidirectional generation system by training only LoRA adapters and a lightweight text head. FullFlow preserves the continuous image flow while incorporating a discrete text insertion mechanism, and introduces dual-timestep inference with a two-dimensional trajectory selection strategy. Without retraining the backbone or making substantial architectural modifications, the method reduces the text-to-image FID on SD3 from 62.7 to 31.6 and improves image-to-text CIDEr from 2.0 to 99.4. Training involves only ~5% of the model parameters, completes within 24 hours, reduces GPU memory usage by 55%, and achieves an 8× throughput gain. The approach also successfully transfers to FLUX.1-dev and visual question answering tasks.

Abstract

Modern text-to-image diffusion models encode rich visual priors, but expose them only through one-way text-conditioned generation. Existing unified vision--language models derived from them recover bidirectional capability through large-scale joint pretraining or substantial retraining of the text pathway, discarding the strong image prior the text-to-image backbone already encodes. We introduce \emph{FullFlow}, a parameter-efficient recipe that upgrades a pretrained rectified-flow text-to-image model into a bidirectional vision--language generator by training only LoRA adapters and lightweight text heads. FullFlow keeps images in their native continuous flow and adds a discrete insertion process for text. Separate image and text timesteps turn inference into trajectory selection in a two-dimensional generative space, enabling text$\rightarrow$image, image$\rightarrow$text, joint sampling, and partial-text prediction with a single backbone. On Stable Diffusion 3 (SD3) under an identical trainable-parameter count and matched LoRA rank, FullFlow improves text$\rightarrow$image FID from $62.7$ to $31.6$ and image$\rightarrow$text CIDEr from $2.0$ to $99.4$ over a LoRA equivalent following the previous SOTA formulation (Dual Diffusion) at matched wall-clock training time, while reducing peak VRAM from ${\sim}84$\,GB to ${\sim}38$\,GB and raising throughput by ${\sim}8\times$ on two RTX A5000 GPUs in under 24 hours, training only ${\sim}5\%$ of the backbone parameters. The same recipe transfers to FLUX.1-dev and supports downstream VQA through partial-text generation. These results show that strong bidirectional vision--language capability can be unlocked from pretrained text-to-image flow models without full multimodal pretraining.

Problem

• bidirectional vision-language generation
• text-to-image models
• flow matching
• parameter-efficient adaptation
• multimodal generation

Innovation

• FullFlow
• rectified flow
• bidirectional vision-language generation
• parameter-efficient adaptation
• LoRA