Representations of Fact, Fiction and Forecast in Large Language Models: Epistemics and Attitudes

2025-06-02

AI Summary

Current large language models (LLMs) struggle to reliably distinguish factual, fictional, and predictive content in uncertain real-world scenarios; their deterministic assertions often lack epistemic grounding, exposing systemic deficiencies in modeling implicit uncertainty. Method: This work introduces the first theoretical framework grounded in cognitive linguistics—specifically, construal-level modality typology—and systematically evaluates mainstream LLMs’ ability to align evidential strength with epistemic stance via controlled narrative experiments and quantitative generation analysis. Results: LLMs exhibit fragility and inconsistency in modality expression, failing to develop robust, semantically grounded representations of uncertainty. The study reveals a fundamental epistemological limitation in contemporary LLMs and establishes a novel methodological foundation for modeling linguistic uncertainty in trustworthy AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Rational speakers are supposed to know what they know and what they do not know, and to generate expressions matching the strength of evidence. In contrast, it is still a challenge for current large language models to generate corresponding utterances based on the assessment of facts and confidence in an uncertain real-world environment. While it has recently become popular to estimate and calibrate confidence of LLMs with verbalized uncertainty, what is lacking is a careful examination of the linguistic knowledge of uncertainty encoded in the latent space of LLMs. In this paper, we draw on typological frameworks of epistemic expressions to evaluate LLMs' knowledge of epistemic modality, using controlled stories. Our experiments show that the performance of LLMs in generating epistemic expressions is limited and not robust, and hence the expressions of uncertainty generated by LLMs are not always reliable. To build uncertainty-aware LLMs, it is necessary to enrich semantic knowledge of epistemic modality in LLMs.

Problem

• Assessing LLMs' ability to generate utterances based on fact confidence
• Evaluating linguistic knowledge of uncertainty in LLMs' latent space
• Improving reliability of uncertainty expressions in large language models

Innovation

• Evaluate LLMs' knowledge using typological frameworks
• Assess epistemic modality with controlled stories
• Enrich semantic knowledge for uncertainty-aware LLMs