Ratio Covers of Convex Sets and Optimal Mixture Density Estimation

2026-02-17

AI Summary

This study addresses the problem of Kullback–Leibler divergence–optimal density estimation from a finite dictionary of densities, encompassing both model aggregation and mixture density estimation, without assuming bounded density ratios or uniform support. By introducing the “Optimal Proportion Covering Theorem”—which characterizes how compact convex sets in the positive orthant can be controlled by exponentially small coordinate subsets—and establishing a distribution-free upper bound on local Hellinger entropy, the work overcomes classical limitations. Integrating tools from information theory, convex geometry, covering number theory, and high-probability deviation analysis, it derives optimal high-probability convergence rates that match known lower bounds in the discrete setting. Furthermore, it yields a novel estimate on the cardinality of ε-Pareto sets for convex attainable sets in multi-objective optimization.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We study density estimation in Kullback-Leibler divergence: given an i.i.d. sample from an unknown density $p$, the goal is to construct an estimator $\widehat p$ such that $\mathrm{KL}(p,\widehat p)$ is small with high probability. We consider two settings involving a finite dictionary of $M$ densities: (i) model aggregation, where $p$ belongs to the dictionary, and (ii) convex aggregation (mixture density estimation), where $p$ is a mixture of densities from the dictionary. Crucially, we make no assumption on the base densities: their ratios may be unbounded and their supports may differ. For both problems, we identify the best possible high-probability guarantees in terms of the dictionary size, sample size, and confidence level. These optimal rates are higher than those achievable when density ratios are bounded by absolute constants; for mixture density estimation, they match existing lower bounds in the special case of discrete distributions. Our analysis of the mixture case hinges on two new covering results. First, we provide a sharp, distribution-free upper bound on the local Hellinger entropy of the class of mixtures of $M$ distributions. Second, we prove an optimal ratio covering theorem for convex sets: for every convex compact set $K\subset \mathbb{R}_+^d$, there exists a subset $A\subset K$ with at most $2^{8d}$ elements such that each element of $K$ is coordinate-wise dominated by an element of $A$ up to a universal constant factor. This geometric result is of independent interest; notably, it yields new cardinality estimates for $\varepsilon$-approximate Pareto sets in multi-objective optimization when the attainable set of objective vectors is convex.

Problem

• density estimation
• Kullback-Leibler divergence
• convex aggregation
• mixture models
• ratio covers

Innovation

• ratio covering
• mixture density estimation
• Hellinger entropy
• convex aggregation
• Pareto set approximation