Two-stage Risk Control with Application to Ranked Retrieval

2024-04-27

AI Summary

To address the challenges of quantifying prediction uncertainty and controlling risk in multi-stage recommender systems (e.g., retrieval-then-ranking cascades), this paper proposes a two-stage risk control framework. First, a predictive model is learned via Learn-then-Test (LTT); second, Conformal Risk Control (CRC) constructs sequential confidence sets satisfying user-specified risk constraints—such as false discovery rate (FDR) ≤ 0.1. Leveraging the inherent sequential structure of cascaded recommenders, our approach introduces a lightweight, theoretically grounded risk control mechanism—the first of its kind for such architectures. Moreover, we design a ranking-aware loss function that jointly optimizes ranking quality and tightens risk bounds. Experiments on MSLR-Web and Yahoo LTRC demonstrate that our method significantly reduces computational overhead while strictly guaranteeing risk controllability under the prescribed FDR constraint.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Sequential Decision Making
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Practical machine learning systems often operate in multiple sequential stages, as seen in ranking and recommendation systems, which typically include a retrieval phase followed by a ranking phase. Effectively assessing prediction uncertainty and ensuring effective risk control in such systems pose significant challenges due to their inherent complexity. To address these challenges, we developed two-stage risk control methods based on the recently proposed learn-then-test (LTT) and conformal risk control (CRC) frameworks. Unlike the methods in prior work that address multiple risks, our approach leverages the sequential nature of the problem, resulting in reduced computational burden. We provide theoretical guarantees for our proposed methods and design novel loss functions tailored for ranked retrieval tasks. The effectiveness of our approach is validated through experiments on two large-scale, widely-used datasets: MSLR-Web and Yahoo LTRC.

Problem

• Machine Learning
• Recommendation Systems
• Accuracy and Risk Assessment

Innovation

• Two-Stage Risk Control
• Machine Learning Systems
• Recommendation Systems