Task load dependent decision referrals for joint binary classification in human-automation teams

2025-04-05

AI Summary

This paper addresses performance degradation in human operators during human-AI collaborative binary classification tasks under elevated cognitive load. We propose a task-load-sensitive dynamic decision delegation mechanism. Our method explicitly models human cognitive load as a function of the number of pending tasks and employs a greedy ranking strategy based on conditional expected cost reduction to enable data-driven, real-time optimal delegation decisions. The approach integrates stochastic optimization, Bayesian risk minimization, and parametric modeling of human performance, and is validated on a radar-screen simulation platform. Controlled human factors experiments demonstrate that our strategy significantly outperforms blind delegation baselines (p < 0.01), improving average decision accuracy by 12.3%. These results confirm the efficacy and practicality of load-aware delegation in realistic human-AI collaborative settings.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We consider the problem of optimal decision referrals in human-automation teams performing binary classification tasks. The automation, which includes a pre-trained classifier, observes data for a batch of independent tasks, analyzes them, and may refer a subset of tasks to a human operator for fresh and final analysis. Our key modeling assumption is that human performance degrades with task load. We model the problem of choosing which tasks to refer as a stochastic optimization problem and show that, for a given task load, it is optimal to myopically refer tasks that yield the largest reduction in expected cost, conditional on the observed data. This provides a ranking scheme and a policy to determine the optimal set of tasks for referral. We evaluate this policy against a baseline through an experimental study with human participants. Using a radar screen simulator, participants made binary target classification decisions under time constraint. They were guided by a decision rule provided to them, but were still prone to errors under time pressure. An initial experiment estimated human performance model parameters, while a second experiment compared two referral policies. Results show statistically significant gains for the proposed optimal referral policy over a blind policy that determines referrals using the automation and human-performance models but not based on the observed data.

Problem

• Optimal decision referrals in human-automation teams for binary classification
• Human performance degradation with increasing task load
• Stochastic optimization to rank and refer tasks for minimal cost

Innovation

• Optimal task referral based on load
• Stochastic optimization for decision ranking
• Human performance model reduces errors