Learning to Trust: How Humans Mentally Recalibrate AI Confidence Signals

2026-03-23

AI Summary

This study addresses the challenge humans face in appropriately relying on AI due to systematic biases in AI confidence signals, such as over- or under-confidence. Through an experiment involving 200 participants engaging in 50 repeated interactions with an AI system, combined with computational modeling, the research reveals that humans dynamically update their baseline trust and sensitivity to AI confidence using asymmetric learning rates. The proposed model, integrating a linear-log-odds (LLO) transformation with the Rescorla–Wagner learning rule, successfully accounts for observed behavioral patterns. Results demonstrate significant improvements in participants’ prediction accuracy, discriminability, and calibration consistency. However, cognitive limitations emerge when confronted with non-monotonic or otherwise counterintuitive confidence mappings, highlighting both the adaptive capacity and inherent constraints of human psychological calibration mechanisms in human–AI collaboration.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Productive human-AI collaboration requires appropriate reliance, yet contemporary AI systems are often miscalibrated, exhibiting systematic overconfidence or underconfidence. We investigate whether humans can learn to mentally recalibrate AI confidence signals through repeated experience. In a behavioral experiment (N = 200), participants predicted the AI's correctness across four AI calibration conditions: standard, overconfidence, underconfidence, and a counterintuitive "reverse confidence" mapping. Results demonstrate robust learning across all conditions, with participants significantly improving their accuracy, discrimination, and calibration alignment over 50 trials. We present a computational model utilizing a linear-in-log-odds (LLO) transformation and a Rescorla-Wagner learning rule to explain these dynamics. The model reveals that humans adapt by updating their baseline trust and confidence sensitivity, using asymmetric learning rates to prioritize the most informative errors. While humans can compensate for monotonic miscalibration, we identify a significant boundary in the reverse confidence scenario, where a substantial proportion of participants struggled to override initial inductive biases. These findings provide a mechanistic account of how humans adapt their trust in AI confidence signals through experience.

Problem

• AI confidence
• trust calibration
• human-AI collaboration
• mental recalibration
• miscalibration

Innovation

• mental recalibration
• AI confidence
• computational modeling
• asymmetric learning rates
• trust adaptation