Do we really need the Rademacher complexities?

2025-02-21

AI Summary

This paper addresses the sample complexity of square-loss learning over convex function classes, challenging the conventional Rademacher-complexity-based analysis paradigm. Methodologically, it integrates optimal mean estimation for real-valued random variables with Talagrand’s generic chaining technique to devise a novel algorithm that bypasses Rademacher complexity entirely. The key contribution is a universal characterization: sample complexity is fundamentally governed by the geometric structure—specifically, the metric entropy under the $L_2$ distance—of the associated limiting Gaussian process, not by Rademacher complexity; consequently, any two learning problems sharing the same $L_2$ metric structure (even under heavy-tailed distributions) exhibit identical sample complexity. Under minimal assumptions, the derived upper bound is tighter and more interpretable than prior results. This work establishes a foundational new perspective for convex learning theory, shifting emphasis from combinatorial complexity measures to intrinsic geometric properties of the hypothesis class.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study the fundamental problem of learning with respect to the squared loss in a convex class. The state-of-the-art sample complexity estimates in this setting rely on Rademacher complexities, which are generally difficult to control. We prove that, contrary to prevailing belief and under minimal assumptions, the sample complexity is not governed by the Rademacher complexities but rather by the behaviour of the limiting gaussian process. In particular, all such learning problems that have the same $L_2$-structure -- even those with heavy-tailed distributions -- share the same sample complexity. This constitutes the first universality result for general convex learning problems. The proof is based on a novel learning procedure, and its performance is studied by combining optimal mean estimation techniques for real-valued random variables with Talagrand's generic chaining method.

Problem

• Study learning with squared loss
• Challenge Rademacher complexities role
• Prove universality in sample complexity

Innovation

• Gaussian process analysis
• Optimal mean estimation
• Generic chaining method