MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation

2026-06-24

AI Summary

This work addresses the limitations of conventional normalizing flows, which are constrained by strict invertibility and struggle to capture high-level semantic structures, as well as the lack of end-to-end integration between masked image modeling (MIM) and generative models. The authors propose MIMFlow, a novel framework that deeply integrates MIM with normalizing flows for the first time. It employs a VAE encoder to infer semantic latent variables from masked images, uses a normalizing flow to model the low-frequency semantic manifold, and deploys a dedicated decoder to synthesize high-frequency details. This design explicitly disentangles semantic and pixel-level generation, enabling end-to-end joint optimization. By overcoming the representational bottleneck of traditional flows, MIMFlow achieves a FID of 2.50 and 71.3% linear probe accuracy on ImageNet 256×256 using only 128 tokens—outperforming comparable flow-based models by 32.8%.

Abstract

Normalizing Flows (NFs) are powerful generative models capable of exact density estimation and sampling. However, their strict invertibility often forces the model to exhaust its capacity on low-level pixel details, hindering the capture of high-level semantic structures. While Masked Image Modeling (MIM) has excelled in representation learning, its integration into generative pipelines has remained largely modular and disjointed. In this paper, we propose MIMFlow, a unified end-to-end framework that jointly optimizes latent semantics, pixel reconstruction, and generative flow. By employing a VAE encoder to infer semantic latent from masked images, MIMFlow achieves a principled decoupling of the generative task: the Normalizing Flow focuses on modeling a simplified, low-frequency semantic manifold, while a specialized decoder handles high-frequency synthesis. This design effectively resolves the inherent capacity bottleneck of NFs, allowing the model to prioritize global structural coherence over redundant noise. Empirical results on ImageNet 256$\times$256 show that MIMFlow-L reaches 71.3\% linear probing accuracy and an FID of 2.50. Despite using only 128 tokens (50\% fewer than standard models), it yields a 32.8\% performance gain over similar-scale NF baselines. Our code is available at https://github.com/MCG-NJU/MIMFlow.

Problem

• Normalizing Flows
• Masked Image Modeling
• image generation
• semantic representation
• generative modeling

Innovation

• Masked Image Modeling
• Normalizing Flows
• End-to-End Generation
• Semantic Decoupling
• Latent Representation