Self-conditioned Flow Map Language Models via Fixed-point Flows

2026-07-01

AI Summary

This work addresses the limitations of existing self-conditioned continuous flow language models, which suffer from suboptimal performance in few-step generation and lack theoretical grounding for their mechanisms. We propose the Fixed-Point Flow (FPF) framework, which formally casts self-conditioning as a fixed-point iteration process and introduces a novel two-dimensional flow architecture that jointly compresses both flow evolution and iteration steps. Building upon this framework, we develop FMLM*, a distillable fixed-point flow language model that unifies generation optimization through fixed-point distillation and flow-mapping distillation. Experimental results on OpenWebText demonstrate that FMLM* significantly outperforms state-of-the-art self-conditioned and few-step generative models, achieving particularly strong performance in single-step and extremely few-step generation settings.

Abstract

Self-conditioning is a core technique that enhances continuous flow-based language models, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow language models with self-conditioning solve a fixed-point iteration that bootstraps the performance of the learned denoiser. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and the flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map language model, FMLM$^\star$, outperforms state-of-the-art self-conditioned models and few-step models in one- and few-step generation on OpenWebText. Code is available at https://github.com/Ugness/self-conditioned-fmlm.

Problem

• self-conditioning
• flow-based language models
• few-step generation
• flow maps
• denoising

Innovation

• fixed-point flows
• self-conditioning
• flow map distillation
• few-step generation
• language modeling