CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

2026-04-29

AI Summary

This study addresses the persistent challenge that explainable AI (XAI) often fails to effectively support human decision-making due to poor user comprehension. To bridge this gap, the work integrates cognitive modeling with user studies to formally represent—within a computationally tractable framework—the reasoning strategies humans employ when interacting with different XAI methods in structured data tasks. Through formative and summative user experiments, feature attribution analyses, and behavioral alignment evaluations, the resulting cognitive model demonstrates significantly greater accuracy than conventional machine learning surrogates in capturing human forward-simulation decision behavior. Beyond elucidating which XAI mechanisms genuinely aid human judgment, the model offers an empirically grounded foundation for designing more effective XAI systems and serves as a high-fidelity, efficient proxy for human-subject experimentation in XAI research.

Abstract

Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.

Problem

• Explainable AI
• human understanding
• cognitive modeling
• reasoning strategies
• forward simulation

Innovation

• Explainable AI
• cognitive modeling
• user study
• feature attribution
• forward simulation