Phase-aware Training Schedule Simplifies Learning in Flow-Based Generative Models

2024-12-10

AI Summary

Training high-dimensional flow-based generative models faces optimization difficulties due to conflated learning phases—specifically, simultaneous estimation of mode probabilities and variances—leading to unstable dynamics. Method: We theoretically establish, for the first time, that autoencoders in Gaussian mixture modeling inherently exhibit two separable learning phases: probability estimation followed by variance refinement. To exploit this structure, we propose a phase-aware temporal dilation scheduling mechanism that dynamically focuses optimization on stage-critical parameters, enabling structural decoupling of training. Our approach integrates a two-level autoencoder-parameterized flow model, velocity field modeling, multi-scale temporal analysis, and feature-level sensitivity detection. Contribution/Results: We formally characterize the staged learning dynamics; empirical evaluation on real-world datasets confirms the efficacy of feature-adaptive training scheduling, yielding substantial improvements in both training efficiency and generation fidelity.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We analyze the training of a two-layer autoencoder used to parameterize a flow-based generative model for sampling from a high-dimensional Gaussian mixture. Previous work shows that the phase where the relative probability between the modes is learned disappears as the dimension goes to infinity without an appropriate time schedule. We introduce a time dilation that solves this problem. This enables us to characterize the learned velocity field, finding a first phase where the probability of each mode is learned and a second phase where the variance of each mode is learned. We find that the autoencoder representing the velocity field learns to simplify by estimating only the parameters relevant to each phase. Turning to real data, we propose a method that, for a given feature, finds intervals of time where training improves accuracy the most on that feature. Since practitioners take a uniform distribution over training times, our method enables more efficient training. We provide preliminary experiments validating this approach.

Problem

• Autoencoders
• High-Dimensionality
• Learning Complexity

Innovation

• Phase-sensitive Training Schedule
• Two-layer Autoencoder
• Dynamic Adjustment for Dimensionality Handling