A Theory of Universal Agnostic Learning

2026-01-28

AI Summary

This work investigates the optimal universal convergence rates for binary classification in the agnostic setting, dispensing with the classical realizability assumption. By integrating tools from statistical learning theory, combinatorial complexity, and concentration inequalities, it establishes the first “tetrachotomy” theorem for agnostic learning, demonstrating that the optimal excess risk convergence rate for any concept class must fall into one of four categories: $e^{-n}$, $e^{-o(n)}$, $o(n^{-1/2})$, or arbitrarily slow. This framework fully characterizes the spectrum of achievable rates in agnostic learning and provides a decision mechanism grounded in a succinct combinatorial structure, thereby laying the theoretical foundation for universal convergence rates in this setting.

Technology Category

• Machine Learning: Learning Theory
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

We provide a complete theory of optimal universal rates for binary classification in the agnostic setting. This extends the realizable-case theory of Bousquet, Hanneke, Moran, van Handel, and Yehudayoff (2021) by removing the realizability assumption on the distribution. We identify a fundamental tetrachotomy of optimal rates: for every concept class, the optimal universal rate of convergence of the excess error rate is one of $e^{-n}$, $e^{-o(n)}$, $o(n^{-1/2})$, or arbitrarily slow. We further identify simple combinatorial structures which determine which of these categories any given concept class falls into.

Problem

• agnostic learning
• universal rates
• binary classification
• excess error rate
• convergence rate

Innovation

• agnostic learning
• universal rates
• binary classification
• combinatorial characterization
• excess error rate