Unexploited Information Value in Human-AI Collaboration

2024-11-03

arXiv.org

AI Summary

In human-AI collaborative decision-making, the informational value contributed by humans and AI is often underestimated or redundantly exploited, limiting overall system efficacy. Method: Grounded in statistical decision theory, this paper introduces the “untapped information value” (UIV) framework—the first of its kind—to quantify feature-level information contributions and systematically characterize complementarity and redundancy between human and AI information utilization. Through experiments on video deepfake detection and human behavioral analysis, we identify differential information value across seven video-level features. Contribution/Results: The analysis reveals critical diagnostic cues that AI exploits but humans consistently overlook. Guided by UIV, we design targeted decision aids that preserve human agency while significantly improving collaborative accuracy (+12.3%) and inter-rater consistency (Cohen’s κ +0.28). The framework provides an interpretable, generalizable theoretical foundation and methodological toolkit for optimizing information use in human-AI collaboration.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Reasoning under Uncertainty: Decision/Utility Theory
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.

Problem

• Human-AI Collaboration
• Information Value
• Decision Making

Innovation

• Mathematical Modeling
• Human-Robot Collaboration
• Information Value in Decision Making