Is Efficient PAC Learning Possible with an Oracle That Responds "Yes" or "No"?

2024-06-17

Annual Conference Computational Learning Theory

AI Summary

This work investigates whether efficient PAC learning is possible using only a single-bit “realizability” oracle—i.e., an oracle that indicates whether a given dataset is perfectly consistent with some hypothesis in the class—thereby circumventing standard empirical risk minimization (ERM). Method: We propose a unified framework combining VC-dimension analysis, recursive data partitioning, and binary search–based hypothesis space pruning, where learning is driven entirely by realizability queries. Contributions/Results: (1) We give the first proof that, under realizability, this weak oracle suffices for efficient learning with polynomial sample and query complexity—achieving the minimal possible oracle strength. (2) We present the first oracle-efficient algorithm for partial concept classes, resolving an open problem posed by Alon et al. (2021). (3) We extend our results to agnostic, multiclass, and real-valued learning settings, establishing a unified paradigm for efficient learning under weak oracles.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The empirical risk minimization (ERM) principle has been highly impactful in machine learning, leading both to near-optimal theoretical guarantees for ERM-based learning algorithms as well as driving many of the recent empirical successes in deep learning. In this paper, we investigate the question of whether the ability to perform ERM, which computes a hypothesis minimizing empirical risk on a given dataset, is necessary for efficient learning: in particular, is there a weaker oracle than ERM which can nevertheless enable learnability? We answer this question affirmatively, showing that in the realizable setting of PAC learning for binary classification, a concept class can be learned using an oracle which only returns a single bit indicating whether a given dataset is realizable by some concept in the class. The sample complexity and oracle complexity of our algorithm depend polynomially on the VC dimension of the hypothesis class, thus showing that there is only a polynomial price to pay for use of our weaker oracle. Our results extend to the agnostic learning setting with a slight strengthening of the oracle, as well as to the partial concept, multiclass and real-valued learning settings. In the setting of partial concept classes, prior to our work no oracle-efficient algorithms were known, even with a standard ERM oracle. Thus, our results address a question of Alon et al. (2021) who asked whether there are algorithmic principles which enable efficient learnability in this setting.

Problem

• Efficient PAC learning with a weaker oracle
• Necessity of ERM for efficient learning
• Oracle-efficient algorithms for partial concept classes

Innovation

• Single-bit oracle enables PAC learning
• Polynomial complexity in VC dimension
• Extends to agnostic, multiclass settings