Metacognition and Confidence Dynamics in Advice Taking from Generative AI

2025-10-30

AI Summary

This study investigates metacognitive dynamics in human–generative-AI interaction—specifically, how users prospectively assess the accuracy of both themselves and AI, and subsequently update their confidence (retrospective confidence) after adopting AI suggestions. Method: Using a text generation task, we employed an experimental design combining voluntary AI request with randomized AI suggestion assignment. Text similarity metrics quantified reliance on AI, while dynamic confidence ratings tracked real-time metacognitive shifts. Contribution/Results: (1) Active adoption of AI suggestions significantly increased retrospective confidence in AI; (2) passive exposure to AI suggestions causally enhanced both AI-directed and self-directed confidence; (3) adopters produced more elaborate but less rigorously verified responses, exposing accuracy blind spots and overreliance risks. This work provides the first empirical evidence of bidirectional regulation between prospective and retrospective confidence—a foundational insight for human-AI interaction design, trustworthy AI systems, and cognitive ergonomics.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Generative Artificial Intelligence (GenAI) can aid humans in a wide range of tasks, but its effectiveness critically depends on users being able to evaluate the accuracy of GenAI outputs and their own expertise. Here we asked how confidence in self and GenAI contributes to decisions to seek and rely on advice from GenAI ('prospective confidence'), and how advice-taking in turn shapes this confidence ('retrospective confidence'). In a novel paradigm involving text generation, participants formulated plans for events, and could request advice from a GenAI (Study 1; N=200) or were randomly assigned to receive advice (Study 2; N=300), which they could rely on or ignore. Advice requests in Study 1 were related to higher prospective confidence in GenAI and lower confidence in self. Advice-seekers showed increased retrospective confidence in GenAI, while those who declined advice showed increased confidence in self. Random assignment in Study 2 revealed that advice exposure increases confidence in GenAI and in self, suggesting that GenAI advice-taking causally boosts retrospective confidence. These results were mirrored in advice reliance, operationalised as the textual similarity between GenAI advice and participants' responses, with reliance associated with increased retrospective confidence in both GenAI and self. Critically, participants who chose to obtain/rely on advice provided more detailed responses (likely due to the output's verbosity), but failed to check the output thoroughly, missing key information. These findings underscore a key role for confidence in interactions with GenAI, shaped by both prior beliefs about oneself and the reliability of AI, and context-dependent exposure to advice.

Problem

• Examining how confidence affects seeking and using AI advice
• Studying how advice-taking shapes confidence in self and AI
• Investigating cognitive biases in evaluating generative AI outputs

Innovation

• Studied metacognition dynamics in human-AI advice interaction
• Used text generation tasks with optional AI advice
• Measured confidence changes through advice seeking and reliance