Human Decision-Making with Persuasive and Narrative LLM Explanations

2026-05-22

AI Summary

This study investigates the impact of narrative explanations generated by large language models (LLMs) on human decision accuracy in classification tasks, with a particular focus on whether persuasive explanations genuinely improve objective performance. Through large-scale behavioral experiments, the research systematically evaluates how LLM-generated narratives of varying persuasiveness influence human decision accuracy, reliance on AI, and the ability to discern AI prediction correctness. The findings reveal that while highly persuasive narratives significantly increase users’ dependence on AI, they do not enhance decision accuracy; instead, they prolong response times and impair users’ capacity to distinguish between correct and incorrect AI predictions. This work uncovers a critical trade-off between the persuasiveness of AI-provided explanations and their efficacy in supporting effective human–AI collaboration.

Abstract

Large language models (LLMs) have the potential to aid and improve human decision-making in classification tasks, not only by providing fairly accurate predictions, but also in their ability to generate cogent narrative explanations of those predictions. Prior work has demonstrated that people generally find AI narrative explanations to be understandable, trustworthy, and convincing for changing beliefs and opinions; however, less is known about the impact of narrative explanations on objective human decision-making performance. Here we conduct a large-scale human behavioral experiment to evaluate decision-making performance with LLM-generated narrative explanations of varying persuasiveness. We found the degree of persuasiveness, or lack thereof, for LLM-based explanations did not meaningfully impact decision accuracy over a simple AI prediction alone, in agreement with typical results with explainable AI based on feature importance. We found evidence that narratives increased reliance on AI, but both when the AI prediction was correct and incorrect. Exploratory analyses also indicated that the more persuasive narratives may have had a detrimental effect on decision response times and the ability to discriminate between a correct and incorrect AI prediction. Overall, this work indicates that including narrative explanations with AI predictions may involve tradeoffs for decision-making performance, and more work is needed to determine how and when narrative explanations impact human decision-making.

Problem

• human decision-making
• narrative explanations
• LLM persuasiveness
• AI reliance
• decision performance

Innovation

• narrative explanations
• large language models
• human decision-making
• persuasiveness
• explainable AI