On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows

2024-02-09

International Conference on Machine Learning

AI Summary

This study addresses the limited expressiveness and theory-practice gap of normalizing flows arising from architectural constraints. Methodologically, it integrates distributional theory analysis, decoupled flow architecture modeling, and function approximation arguments under condition-number constraints. Theoretically, it establishes, for the first time under well-conditioned neural network assumptions, the universal distributional approximation capability of coupling-based flows (e.g., RealNVP); concurrently, it proves the intrinsic non-universality of volume-preserving flows—demonstrating they can only model specific distribution subclasses—and proposes a repair mechanism based on invertible perturbations and conditional volume scaling. These results bridge the gap between theoretical expectations and empirical performance, providing a new paradigm for designing normalizing flow models that simultaneously guarantee theoretical soundness and practical efficacy.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Security and Privacy: Large-scale security measurements

Abstract

We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.

Problem

• Standardization Streams
• Computational Challenges
• Enhancement Methods

Innovation

• Standardizing Flows
• Volume-Preserving Limitations
• Affine Coupling Superiority