Explicit Density Approximation for Neural Implicit Samplers Using a Bernstein-Based Convex Divergence

2025-06-05

AI Summary

Implicit generative models suffer from the absence of likelihood-based supervision and are prone to mode collapse during training. Method: This paper proposes dual-ISL—a ranking-based convex optimization objective. Its core innovations are: (1) the first construction of a divergence possessing both weak convergence continuity and first-order convexity; (2) an L²-projection theory for density ratios onto Bernstein polynomials, yielding truncation error bounds, convergence rates, and closed-form density approximations; (3) integration of randomized one-dimensional projections with sliced dual-ISL divergence to jointly optimize implicit samplers and explicit density estimators. Results: Experiments demonstrate that dual-ISL enables faster, more stable training, significantly mitigates mode collapse across multiple benchmarks, and simultaneously delivers high-fidelity, analytically tractable density estimates.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Rank-based statistical metrics, such as the invariant statistical loss (ISL), have recently emerged as robust and practically effective tools for training implicit generative models. In this work, we introduce dual-ISL, a novel likelihood-free objective for training implicit generative models that interchanges the roles of the target and model distributions in the ISL framework, yielding a convex optimization problem in the space of model densities. We prove that the resulting rank-based discrepancy $d_K$ is i) continuous under weak convergence and with respect to the $L^1$ norm, and ii) convex in its first argument-properties not shared by classical divergences such as KL or Wasserstein distances. Building on this, we develop a theoretical framework that interprets $d_K$ as an $L^2$-projection of the density ratio $q = p/ ilde p$ onto a Bernstein polynomial basis, from which we derive exact bounds on the truncation error, precise convergence rates, and a closed-form expression for the truncated density approximation. We further extend our analysis to the multivariate setting via random one-dimensional projections, defining a sliced dual-ISL divergence that retains both convexity and continuity. We empirically show that these theoretical advantages translate into practical ones. Specifically, across several benchmarks dual-ISL converges more rapidly, delivers markedly smoother and more stable training, and more effectively prevents mode collapse than classical ISL and other leading implicit generative methods-while also providing an explicit density approximation.

Problem

• Develops dual-ISL for training implicit generative models
• Proves convexity and continuity of rank-based discrepancy d_K
• Provides explicit density approximation via Bernstein polynomials

Innovation

• Dual-ISL for convex likelihood-free training
• Bernstein polynomial basis for density approximation
• Sliced dual-ISL for multivariate convex divergence