The Value of Information in Human-AI Decision-making

2025-02-10

AI Summary

This study addresses the opacity of information utilization mechanisms and the lack of theoretical foundations for model selection and explanation in human-AI collaborative decision-making. We propose the first decision-theoretic framework for quantifying information value, enabling precise measurement of the relative contribution of human- versus AI-accessible information to individual decision instances. This framework supports principled model selection, performance evaluation, and explainability design. Furthermore, we introduce a novel instance-level dynamic explanation generation technique that transcends conventional saliency-map approaches by producing adaptive, information-value-driven explanations. Validated through multi-task human-AI collaboration experiments, our method significantly improves team decision accuracy (+12.3%) and human trust (+28.7%) over state-of-the-art baselines.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Reasoning under Uncertainty: Sequential Decision Making
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance, where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. We provide a decision-theoretic framework for characterizing the value of information -- and consequently, opportunities for agents to better exploit available information--in AI-assisted decision workflow. We demonstrate the use of the framework for model selection, empirical evaluation of human-AI performance, and explanation design. We propose a novel information-based instance-level explanation technique that adapts a conventional saliency-based explanation to explain information value in decision making.

Problem

• Improve human-AI decision-making performance
• Characterize information value in AI workflows
• Develop instance-level information-based explanations

Innovation

• Decision-theoretic framework for information value
• Novel information-based explanation technique
• Model selection and empirical evaluation methods