Probably Approximately Precision and Recall Learning

2024-11-20

arXiv.org

AI Summary

In multi-label learning and recommendation systems, prevalent positive-unlabeled (PU) feedback—where only positive instances are observed—poses fundamental challenges for learning under “one-sided” labeling. Method: This paper introduces the first PAC learning framework jointly optimizing precision and recall. Departing from standard single-label or fully supervised settings, it employs set-valued functions as the hypothesis class, unifying binary classification, multi-label learning, and partial-label learning. Contribution/Results: We prove that empirical risk minimization (ERM) is inherently inconsistent in this setting. We establish the first precision-recall bi-objective PAC theory, derive optimal sample complexity for the realizable case, and—crucially—provide the first nontrivial multiplicative approximation guarantee for the agnostic case, circumventing known impossibility results for additive error. Leveraging graph-structured representations of user-item interactions, we design a positive-sample-specific risk minimization algorithm, enabling principled learning from one-sided feedback.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Precision and Recall are foundational metrics in machine learning where both accurate predictions and comprehensive coverage are essential, such as in recommender systems and multi-label learning. In these tasks, balancing precision (the proportion of relevant items among those predicted) and recall (the proportion of relevant items successfully predicted) is crucial. A key challenge is that one-sided feedback--where only positive examples are observed during training--is inherent in many practical problems. For instance, in recommender systems like YouTube, training data only consists of videos that a user has actively selected, while unselected items remain unseen. Despite this lack of negative feedback in training, avoiding undesirable recommendations at test time is essential. We introduce a PAC learning framework where each hypothesis is represented by a graph, with edges indicating positive interactions, such as between users and items. This framework subsumes the classical binary and multi-class PAC learning models as well as multi-label learning with partial feedback, where only a single random correct label per example is observed, rather than all correct labels. Our work uncovers a rich statistical and algorithmic landscape, with nuanced boundaries on what can and cannot be learned. Notably, classical methods like Empirical Risk Minimization fail in this setting, even for simple hypothesis classes with only two hypotheses. To address these challenges, we develop novel algorithms that learn exclusively from positive data, effectively minimizing both precision and recall losses. Specifically, in the realizable setting, we design algorithms that achieve optimal sample complexity guarantees. In the agnostic case, we show that it is impossible to achieve additive error guarantees--as is standard in PAC learning--and instead obtain meaningful multiplicative approximations.

Problem

• Addresses learning from one-sided feedback with only positive examples
• Extends PAC framework to set function hypotheses for multi-label tasks
• Develops new algorithms achieving optimal sample complexity and multiplicative guarantees

Innovation

• PAC framework for set function hypotheses
• Algorithms learning from positive data alone
• Multiplicative approximation guarantees in agnostic case