Learning To Defer To A Population With Limited Demonstrations

2025-10-22

AI Summary

Learning-to-Defer (L2D) systems struggle to reliably defer to crowd experts in real-world settings due to severe scarcity of expert-labeled data. Method: We propose a context-aware semi-supervised meta-learning framework that learns expert-specific embeddings from minimal expert demonstrations. These embeddings serve two synergistic purposes: (i) enabling high-quality pseudo-label generation at scale to alleviate annotation bottlenecks, and (ii) supporting plug-and-play test-time adaptation to unseen experts. Contribution/Results: By tightly integrating meta-learning, contextual encoding, and semi-supervised pseudo-labeling, our approach significantly enhances model generalization and deployment flexibility. On three benchmark datasets, it achieves near-oracle performance using only a handful of real expert demonstrations—substantially reducing data dependency and advancing the practical deployment of human-AI collaborative systems.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Learning to Search
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper addresses the critical data scarcity that hinders the practical deployment of learning to defer (L2D) systems to the population. We introduce a context-aware, semi-supervised framework that uses meta-learning to generate expert-specific embeddings from only a few demonstrations. We demonstrate the efficacy of a dual-purpose mechanism, where these embeddings are used first to generate a large corpus of pseudo-labels for training, and subsequently to enable on-the-fly adaptation to new experts at test-time. The experiment results on three different datasets confirm that a model trained on these synthetic labels rapidly approaches oracle-level performance, validating the data efficiency of our approach. By resolving a key training bottleneck, this work makes adaptive L2D systems more practical and scalable, paving the way for human-AI collaboration in real-world environments. To facilitate reproducibility and address implementation details not covered in the main text, we provide our source code and training configurations at https://github.com/nil123532/learning-to-defer-to-a-population-with-limited-demonstrations.

Problem

• Addresses data scarcity in learning to defer systems deployment
• Uses meta-learning to generate expert embeddings from few demonstrations
• Enables on-the-fly adaptation to new experts at test time

Innovation

• Meta-learning generates expert-specific embeddings from few demonstrations
• Dual-purpose mechanism creates pseudo-labels for training
• Enables on-the-fly adaptation to new experts at test-time