Efficient Optimal PAC Learning

2025-02-05

AI Summary

Empirical Risk Minimization (ERM) incurs prohibitively high computational costs in optimal PAC learning, constituting a fundamental bottleneck. Method: We propose the first optimal PAC learner that entirely avoids invoking ERM as a subroutine. Through rigorous theoretical construction and computational complexity analysis within the PAC framework, we decouple computational cost from sample complexity—bypassing deterministic subsampling, Bagging, and other conventional approximation heuristics. Contribution/Results: Our learner establishes a novel computational-statistical trade-off pathway, achieving optimal statistical efficiency—i.e., minimal sample complexity—while provably reducing computational overhead. This breaks the intrinsic computational barrier imposed by ERM and provides a rigorous theoretical foundation for designing efficient, scalable optimal learning algorithms.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Recent advances in the binary classification setting by Hanneke [2016b] and Larsen [2023] have resulted in optimal PAC learners. These learners leverage, respectively, a clever deterministic subsampling scheme and the classic heuristic of bagging Breiman [1996]. Both optimal PAC learners use, as a subroutine, the natural algorithm of empirical risk minimization. Consequently, the computational cost of these optimal PAC learners is tied to that of the empirical risk minimizer algorithm. In this work, we seek to provide an alternative perspective on the computational cost imposed by the link to the empirical risk minimizer algorithm. To this end, we show the existence of an optimal PAC learner, which offers a different tradeoff in terms of the computational cost induced by the empirical risk minimizer.

Problem

• Optimal PAC learners
• computational cost
• empirical risk minimizer

Innovation

• Deterministic subsampling scheme
• Bagging heuristic
• Empirical risk minimization