Flexible Tails for Normalizing Flows

2024-06-22

arXiv.org

AI Summary

Standard normalizing flows struggle to model heavy-tailed distributions, limiting their performance in density estimation and variational inference. To address this, we propose the Tail Transform Flow (TTF), which retains a Gaussian base distribution while decoupling heavy-tailed modeling into a learnable, parameterized tail transformation—such as power-law or piecewise affine mappings—applied solely in the final flow layer. This design avoids the optimization challenges associated with heavy-tailed base distributions. TTF is compatible with mainstream normalizing flow architectures (e.g., RealNVP, Glow) and incorporates gradient-stabilizing training strategies. Experiments demonstrate that TTF significantly outperforms state-of-the-art methods—including TAF—in multivariate heavy-tailed density estimation and variational inference. Notably, it achieves log-likelihood improvements of 0.3–1.1 nats in high-dimensional settings (d ≥ 100) and under extreme heavy tails (tail index α ≤ 2), while exhibiting strong robustness and generalization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

Normalizing flows are a flexible class of probability distributions, expressed as transformations of a simple base distribution. A limitation of standard normalizing flows is representing distributions with heavy tails, which arise in applications to both density estimation and variational inference. A popular current solution to this problem is to use a heavy tailed base distribution. Examples include the tail adaptive flow (TAF) methods of Laszkiewicz et al (2022). We argue this can lead to poor performance due to the difficulty of optimising neural networks, such as normalizing flows, under heavy tailed input. This problem is demonstrated in our paper. We propose an alternative: use a Gaussian base distribution and a final transformation layer which can produce heavy tails. We call this approach tail transform flow (TTF). Experimental results show this approach outperforms current methods, especially when the target distribution has large dimension or tail weight.

Problem

• Standard normalizing flows struggle with heavy-tailed distributions.
• Heavy-tailed base distributions can hinder neural network optimization.
• Proposing tail transform flow for better heavy-tail performance.

Innovation

• Uses Gaussian base distribution for stability
• Final transform layer creates heavy tails
• Outperforms heavy-tailed base distribution methods