Confidence Without Competence in AI-Assisted Knowledge Work

2026-04-10

AI Summary

This study addresses the tendency of large language models to induce user overconfidence and suppress deep thinking in knowledge work. To mitigate this, the authors propose three lightweight interaction mechanisms—future self-explanation, contrastive learning, and guided prompting—and implement them in a web-based system called Deep3. Through a mixed-methods approach combining user interviews and controlled experiments, the research uncovers systematic misalignments among effort, confidence, and learning outcomes in AI-augmented tasks. Findings indicate that future self-explanation most effectively aligns subjective understanding with objective learning gains, while guided prompting yields the greatest improvement in learning without significantly increasing cognitive load or frustration. These results offer a novel pathway for designing human-AI collaboration paradigms that foster deeper cognitive engagement.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large Language Models (LLMs) are widely used by students, yet their tendency to provide fast and complete answers may discourage reflection and foster overconfidence. We examined how alternative LLM interaction designs support deeper thinking without excessively increasing cognitive burden. We conducted a two-phase mixed-methods study. In Phase 1, interviews with 16 Gen Z students informed the design of Deep3, a web-based system with three interaction modes: \emph{a)} future-self explanations, \emph{b)} contrastive learning, and \emph{c)} guided hints. In Phase 2, we evaluated Deep3 with 85 participants across two learning tasks. We found that a standard single-agent baseline produced high perceived understanding despite the lowest objective learning. In contrast, future-self explanations imposed higher cognitive workload yet yielded the closest alignment between perceived and actual understanding, while guided hints achieved the largest learning gains without a proportional increase in frustration. These findings show that effort, confidence, and learning systematically diverge in LLM-supported work.

Problem

• overconfidence
• large language models
• perceived understanding
• actual learning
• cognitive burden

Innovation

• LLM interaction design
• future-self explanations
• guided hints
• cognitive workload
• perceived vs. actual understanding