Tight Bounds on Jensen's Gap: Novel Approach with Applications in Generative Modeling

2025-02-06

AI Summary

This paper addresses the problem of deriving tight upper and lower bounds on the Jensen gap—i.e., $mathbb{E}[f(X)] - f(mathbb{E}[X])$—under varying convex functions $f$ and distributional assumptions. We propose a general, rigorous convex-analytic framework that systematically constructs tight bounds for broad classes of convex functions and distributions; notably, for the logarithmic function under log-normal distributions, we derive the tightest closed-form bounds to date. Our method integrates moment-constrained optimization (Struski et al., 2023), precise convexity characterization, and probabilistic inequalities, validated empirically on synthetic data. Theoretically, we establish a universal paradigm for constructing tight Jensen-gap bounds. Experimentally, when applied to log-likelihood estimation in VAEs and flow-based generative models, our bounds reduce estimation error by 12.7%–23.4% over state-of-the-art methods, significantly enhancing the reliability of generative model evaluation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Among various mathematical tools of particular interest are those that provide a common basis for researchers in different scientific fields. One of them is Jensen's inequality, which states that the expectation of a convex function is greater than or equal to the function evaluated at the expectation. The resulting difference, known as Jensen's gap, became the subject of investigation by both the statistical and machine learning communities. Among many related topics, finding lower and upper bounds on Jensen's gap (under different assumptions on the underlying function and distribution) has recently become a problem of particular interest. In our paper, we take another step in this direction by providing a novel general and mathematically rigorous technique, motivated by the recent results of Struski et al. (2023). In addition, by studying in detail the case of the logarithmic function and the log-normal distribution, we explore a method for tightly estimating the log-likelihood of generative models trained on real-world datasets. Furthermore, we present both analytical and experimental arguments in support of the superiority of our approach in comparison to existing state-of-the-art solutions, contingent upon fulfillment of the criteria set forth by theoretical studies and corresponding experiments on synthetic data.

Problem

• Provides tight bounds on Jensen's gap
• Explores log-likelihood estimation in generative models
• Presents a novel rigorous mathematical technique

Innovation

• Novel bounds on Jensen's gap
• Application in generative modeling
• Analytical and experimental validation