What large language models know and what people think they know

2024-01-24

Nature Machine Intelligence

AI Summary

This work identifies a systematic misalignment between large language models’ (LLMs) actual knowledge boundaries and human subjective perceptions of their capabilities. To quantify this gap, the study employs multi-source factual probing, controlled prompt engineering, and large-scale crowdsourced metacognitive experiments—introducing “cognitive alignment” as a novel evaluation paradigm for assessing LLM–human epistemic congruence. Results reveal that humans overestimate LLM performance in causal reasoning and domain-specific tasks by 37%, while underestimating their pattern memorization capacity. Furthermore, default explanations and explanation length significantly bias users’ confidence in model outputs—even when explanation quality does not improve answer accuracy. Building on these findings, the paper proposes an explainability calibration framework that enhances reliability and decision quality in human–model collaboration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Problem

• Calibration gap in LLM confidence
• Discrimination gap in answer accuracy
• Impact of explanation length on user trust

Innovation

• Calibration gap analysis
• Discrimination gap reduction
• Adjusted LLM explanations