Bayesian Inference for Correlated Human Experts and Classifiers

2025-06-05

AI Summary

This work addresses human-AI collaborative prediction, focusing on minimizing expert query cost while effectively fusing outputs from pretrained classifiers with human expert judgments. We propose a Bayesian framework grounded in a shared latent variable model, the first to represent expert correlations via a common latent representation—enabling posterior inference for unobserved expert labels and simulation-based utility evaluation of queries. The method employs variational inference (or MCMC) to learn the joint distribution over latent variables and integrates an active learning strategy to optimize the sequence of expert queries. Experiments on medical diagnosis tasks, CIFAR-10H, and ImageNet-16H demonstrate that our approach significantly reduces expert query cost—outperforming state-of-the-art baselines by a large margin—while maintaining high predictive accuracy. It unifies uncertainty-aware prediction, multi-source information fusion, and sequential decision-making under limited expert interaction.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Applications of machine learning often involve making predictions based on both model outputs and the opinions of human experts. In this context, we investigate the problem of querying experts for class label predictions, using as few human queries as possible, and leveraging the class probability estimates of pre-trained classifiers. We develop a general Bayesian framework for this problem, modeling expert correlation via a joint latent representation, enabling simulation-based inference about the utility of additional expert queries, as well as inference of posterior distributions over unobserved expert labels. We apply our approach to two real-world medical classification problems, as well as to CIFAR-10H and ImageNet-16H, demonstrating substantial reductions relative to baselines in the cost of querying human experts while maintaining high prediction accuracy.

Problem

• Combine human expert opinions with classifier predictions efficiently
• Model expert correlations using Bayesian framework for label inference
• Reduce human query costs while maintaining high accuracy

Innovation

• Bayesian framework models expert correlation
• Simulation-based inference reduces expert queries
• Joint latent representation enhances label prediction