AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventions

2023-04-18

AI Summary

In AI-assisted decision-making, humans often misjudge the quality of AI recommendations, leading to inappropriate reliance on or override of AI outputs—conflating “reliance behavior” with actual decision quality. Method: This paper introduces the first systematic conceptual decoupling of these constructs, establishing a formal theoretical framework that defines necessary conditions for human-AI complementarity; proposes a visualization-analytic model to uncover differential mechanisms by which explanations and other interventions affect reliance behavior versus decision quality; and validates the theory through integrated conceptual analysis, formal modeling, and meta-level empirical investigation. Contribution/Results: We demonstrate that altering reliance behavior alone does not improve decision quality; genuine complementarity requires three conditions: calibratability, accountability, and contextual adaptability. Our work establishes a new paradigm and evaluation benchmark for designing AI interventions explicitly targeted at achieving authentic decision-quality gains.

Abstract

In AI-assisted decision-making, a central promise of having a human-in-the-loop is that they should be able to complement the AI system by overriding its wrong recommendations. In practice, however, we often see that humans cannot assess the correctness of AI recommendations and, as a result, adhere to wrong or override correct advice. Different ways of relying on AI recommendations have immediate, yet distinct, implications for decision quality. Unfortunately, reliance and decision quality are often inappropriately conflated in the current literature on AI-assisted decision-making. In this work, we disentangle and formalize the relationship between reliance and decision quality, and we characterize the conditions under which human-AI complementarity is achievable. To illustrate how reliance and decision quality relate to one another, we propose a visual framework and demonstrate its usefulness for interpreting empirical findings, including the effects of interventions like explanations. Overall, our research highlights the importance of distinguishing between reliance behavior and decision quality in AI-assisted decision-making.

Problem

• Human ability to override AI recommendations
• Relationship between reliance and decision quality
• Conditions for human-AI complementarity

Innovation

• Human-AI complementarity formalization
• Visual framework for reliance interpretation
• Distinguishing reliance from decision quality