What large language models know and what people think they know
🤖 AI Summary
This work identifies a systematic misalignment between large language models’ (LLMs) actual knowledge boundaries and human subjective perceptions of their capabilities. To quantify this gap, the study employs multi-source factual probing, controlled prompt engineering, and large-scale crowdsourced metacognitive experiments—introducing “cognitive alignment” as a novel evaluation paradigm for assessing LLM–human epistemic congruence. Results reveal that humans overestimate LLM performance in causal reasoning and domain-specific tasks by 37%, while underestimating their pattern memorization capacity. Furthermore, default explanations and explanation length significantly bias users’ confidence in model outputs—even when explanation quality does not improve answer accuracy. Building on these findings, the paper proposes an explainability calibration framework that enhances reliability and decision quality in human–model collaboration.
Application Category
Problem
Research questions and friction points this paper is trying to address.
Calibration gap in LLM confidence
Discrimination gap in answer accuracy
Impact of explanation length on user trust
Innovation
Methods, ideas, or system contributions that make the work stand out.
Calibration gap analysis
Discrimination gap reduction
Adjusted LLM explanations
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
