PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams

2026-05-08

AI Summary

This work addresses the challenge of efficiently integrating human and AI predictions when neither achieves satisfactory performance alone in classification tasks. The authors propose a multi-stage human-AI collaboration framework that innovatively treats humans as deterministic annotators and AI models as probabilistic classifiers. By applying Bayesian inference, the method fuses instance-level and class-level calibrated probabilities to produce more accurate joint predictions. Evaluated across multiple classification benchmarks, the approach significantly improves overall system accuracy while reducing reliance on costly human intervention, thereby achieving a synergistic optimization of performance and cost efficiency.

Abstract

Human-AI teams play a pivotal role in improving overall system performance when neither the human nor the model can achieve such performance on their own. With the advent of powerful and accessible Generative AI models, several mundane tasks have morphed into Human-AI team tasks. From writing essays to developing advanced algorithms, humans have found that using AI assistance has led to an accelerated work pace like never before. In classification tasks, where the final output is a single hard label, it is crucial to address the combination of human and model output. Prior work elegantly solves this problem using Bayes rule, using the assumption that human and model output are conditionally independent given the ground truth. Specifically, it discusses a combination method to combine a single deterministic labeler (the human) and a probabilistic labeler (the classifier model) using the model's instance-level and the human's class-level calibrated probabilities.

Problem

• Human-AI teams
• classification tasks
• output combination
• hard label
• performance improvement

Innovation

• Human-AI collaboration
• cost-effective performance
• calibrated probabilities
• Bayesian combination
• multi-stage framework