Efficient Generative Modeling via Penalized Optimal Transport Network

2024-02-16

AI Summary

Existing generative models (e.g., WGANs) suffer from mode collapse and exhibit unstable Wasserstein distance estimation in high dimensions, leading to inaccurate modeling of tail distributions and sparse modes, as well as inefficient large-scale generation. Method: We propose POTNet, a novel framework introducing the Marginal-Penalized Wasserstein (MPW) distance—a discriminator-free, non-adversarial metric that enables end-to-end direct optimization by penalizing marginal discrepancies. Contribution/Results: Theoretically, we establish the first non-asymptotic generalization bound for MPW loss and derive the convergence rate of the generated distribution to the target. Methodologically, POTNet jointly aligns joint distributions while precisely characterizing heavy tails and minor modes. Experiments demonstrate significant mitigation of mode collapse, superior fidelity in reproducing fine-grained distributional details, and sampling speedups of several orders of magnitude over state-of-the-art methods—enabling efficient large-scale synthetic data generation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Web data generation and simulation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

The generation of synthetic data with distributions that faithfully emulate the underlying data-generating mechanism holds paramount significance. Wasserstein Generative Adversarial Networks (WGANs) have emerged as a prominent tool for this task; however, due to the delicate equilibrium of the minimax formulation and the instability of Wasserstein distance in high dimensions, WGAN often manifests the pathological phenomenon of mode collapse. This results in generated samples that converge to a restricted set of outputs and fail to adequately capture the tail behaviors of the true distribution. Such limitations can lead to serious downstream consequences. To this end, we propose the Penalized Optimal Transport Network (POTNet), a versatile deep generative model based on the marginally-penalized Wasserstein (MPW) distance. Through the MPW distance, POTNet effectively leverages low-dimensional marginal information to guide the overall alignment of joint distributions. Furthermore, our primal-based framework enables direct evaluation of the MPW distance, thus eliminating the need for a critic network. This formulation circumvents training instabilities inherent in adversarial approaches and avoids the need for extensive parameter tuning. We derive a non-asymptotic bound on the generalization error of the MPW loss and establish convergence rates of the generative distribution learned by POTNet. Our theoretical analysis together with extensive empirical evaluations demonstrate the superior performance of POTNet in accurately capturing underlying data structures, including their tail behaviors and minor modalities. Moreover, our model achieves orders of magnitude speedup during the sampling stage compared to state-of-the-art alternatives, which enables computationally efficient large-scale synthetic data generation.

Problem

• Generative Models
• WGANs
• Data Fidelity

Innovation

• POTNet
• MPW distance
• WGANs optimization