$alpha$-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models

2025-04-14

AI Summary

This paper addresses the lack of a unified theoretical foundation for Continuous-State Discrete Flow Matching (CS-DFM) in discrete data generation. We propose a differentiable generative framework grounded in information geometry’s α-representation. First, we introduce the α-Flow family, revealing that CS-DFM fundamentally models Riemannian flows on the probability simplex under distinct α-geometries. Second, we derive a unified variational bound linking flow matching loss to discrete negative log-likelihood. Third, we theoretically identify and empirically validate the superiority of intermediate α ∈ (0,1) geometries—balancing modeling expressivity and optimization stability. Experiments on image, protein sequence, and text generation tasks demonstrate substantial improvements over discrete-state flow matching baselines; notably, our method more accurately captures token-level entropy distributions. This work establishes a new paradigm for discrete generative modeling that unifies theoretical rigor with practical efficacy.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Natural Language Processing: Generation
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Recent efforts have extended the flow-matching framework to discrete generative modeling. One strand of models directly works with the continuous probabilities instead of discrete tokens, which we colloquially refer to as Continuous-State Discrete Flow Matching (CS-DFM). Existing CS-DFM models differ significantly in their representations and geometric assumptions. This work presents a unified framework for CS-DFM models, under which the existing variants can be understood as operating on different $alpha$-representations of probabilities. Building upon the theory of information geometry, we introduce $alpha$-Flow, a family of CS-DFM models that adheres to the canonical $alpha$-geometry of the statistical manifold, and demonstrate its optimality in minimizing the generalized kinetic energy. Theoretically, we show that the flow matching loss for $alpha$-flow establishes a unified variational bound for the discrete negative log-likelihood. We comprehensively evaluate different instantiations of $alpha$-flow on various discrete generation domains to demonstrate their effectiveness in discrete generative modeling, including intermediate values whose geometries have never been explored before. $alpha$-flow significantly outperforms its discrete-state counterpart in image and protein sequence generation and better captures the entropy in language modeling.

Problem

• Unifies Continuous-State Discrete Flow Matching models under one framework
• Introduces α-Flow adhering to α-geometry for optimal kinetic energy minimization
• Demonstrates α-Flow's superiority in discrete generative modeling tasks

Innovation

• Unified framework for Continuous-State Discrete Flow Matching
• Introduces α-Flow with α-geometry adherence
• Optimizes generalized kinetic energy minimization