Uniform Laws of Large Numbers in Product Spaces

2026-03-25

AI Summary

This study investigates the conditions under which the Uniform Law of Large Numbers (ULLN) holds in Cartesian product spaces when the joint distribution is absolutely continuous with respect to the product of marginal distributions. To characterize the complexity of function classes under this product structure, the authors introduce the notion of “linear VC dimension” and establish that the ULLN holds if and only if this dimension is finite. The linear VC dimension refines the classical VC dimension—evidenced by the fact that, for instance, the class of convex sets has linear VC dimension merely equal to two. Moreover, the work demonstrates the failure of the standard empirical mean estimator in this setting and constructs a novel non-standard estimator that achieves uniform convergence.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Uniform laws of large numbers form a cornerstone of Vapnik--Chervonenkis theory, where they are characterized by the finiteness of the VC dimension. In this work, we study uniform convergence phenomena in cartesian product spaces, under assumptions on the underlying distribution that are compatible with the product structure. Specifically, we assume that the distribution is absolutely continuous with respect to the product of its marginals, a condition that captures many natural settings, including product distributions, sparse mixtures of product distributions, distributions with low mutual information, and more. We show that, under this assumption, a uniform law of large numbers holds for a family of events if and only if the linear VC dimension of the family is finite. The linear VC dimension is defined as the maximum size of a shattered set that lies on an axis-parallel line, namely, a set of vectors that agree on all but at most one coordinate. This dimension is always at most the classical VC dimension, yet it can be arbitrarily smaller. For instance, the family of convex sets in $\mathbb{R}^d$ has linear VC dimension $2$, while its VC dimension is infinite already for $d\ge 2$. Our proofs rely on estimator that departs substantially from the standard empirical mean estimator and exhibits more intricate structure. We show that such deviations from the standard empirical mean estimator are unavoidable in this setting. Throughout the paper, we propose several open questions, with a particular focus on quantitative sample complexity bounds.

Problem

• Uniform Laws of Large Numbers
• Product Spaces
• VC Dimension
• Linear VC Dimension
• Absolute Continuity

Innovation

• linear VC dimension
• uniform law of large numbers
• product spaces
• absolute continuity with respect to product measure
• non-standard estimators