Budgeted Multiple-Expert Deferral

2025-10-30

AI Summary

Existing deferred-learning algorithms incur prohibitive computational overhead during training by querying all experts for every sample—contradicting their core principle of “on-demand expert invocation.” This paper introduces a budget-constrained multi-expert delegation framework that, for the first time, explicitly optimizes expert query cost during supervised training: given ground-truth labels, it dynamically selects only a subset of high-information experts per sample to minimize querying overhead while preserving prediction accuracy. The framework encompasses both two-stage and single-stage algorithms, accompanied by theoretical guarantees on generalization error and label complexity. Empirical evaluation across multiple benchmark tasks demonstrates substantial reductions in expert queries—up to 72%—while maintaining or even improving classification accuracy, thereby validating the framework’s efficiency and practical viability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Learning to defer uncertain predictions to costly experts offers a powerful strategy for improving the accuracy and efficiency of machine learning systems. However, standard training procedures for deferral algorithms typically require querying all experts for every training instance, an approach that becomes prohibitively expensive when expert queries incur significant computational or resource costs. This undermines the core goal of deferral: to limit unnecessary expert usage. To overcome this challenge, we introduce the budgeted deferral framework, which aims to train effective deferral algorithms while minimizing expert query costs during training. We propose new algorithms for both two-stage and single-stage multiple-expert deferral settings that selectively query only a subset of experts per training example. While inspired by active learning, our setting is fundamentally different: labels are already known, and the core challenge is to decide which experts to query in order to balance cost and predictive performance. We establish theoretical guarantees for both of our algorithms, including generalization bounds and label complexity analyses. Empirical results across several domains show that our algorithms substantially reduce training costs without sacrificing prediction accuracy, demonstrating the practical value of our budget-aware deferral algorithms.

Problem

• Training deferral algorithms without querying all costly experts
• Minimizing expert query costs during training while maintaining accuracy
• Selectively querying subsets of experts to balance cost and performance

Innovation

• Budgeted deferral framework minimizes expert query costs
• Selective expert querying per training example reduces expenses
• Algorithms maintain accuracy while cutting training costs