On Fact and Frequency: LLM Responses to Misinformation Expressed with Uncertainty

2025-03-06

AI Summary

This study investigates the robustness of large language models (LLMs) in fact-checking claims containing uncertainty expressions. Methodologically, we systematically rewrite statements according to a taxonomy of uncertainty—belief-based, modal, and linguistic cue-based—and conduct multi-model comparative experiments using GPT-4o, Llama3, and DeepSeek-v2. Our results reveal: (1) belief-based uncertainty significantly impairs fact-checking accuracy, causing 25% of originally labeled “false” claims to be misclassified as “not false”—a shift unexplained by other uncertainty types; (2) we uncover, for the first time, a statistically significant weak cross-task correlation (p < 0.01) between LLMs’ factual judgments and their frequency estimates, suggesting shared underlying cognitive mechanisms. These findings provide novel empirical evidence of LLMs’ vulnerability to semantic ambiguity and propose a dual-task evaluation paradigm to advance trustworthy AI assessment.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We study LLM judgments of misinformation expressed with uncertainty. Our experiments study the response of three widely used LLMs (GPT-4o, LlaMA3, DeepSeek-v2) to misinformation propositions that have been verified false and then are transformed into uncertain statements according to an uncertainty typology. Our results show that after transformation, LLMs change their factchecking classification from false to not-false in 25% of the cases. Analysis reveals that the change cannot be explained by predictors to which humans are expected to be sensitive, i.e., modality, linguistic cues, or argumentation strategy. The exception is doxastic transformations, which use linguistic cue phrases such as"It is believed ...".To gain further insight, we prompt the LLM to make another judgment about the transformed misinformation statements that is not related to truth value. Specifically, we study LLM estimates of the frequency with which people make the uncertain statement. We find a small but significant correlation between judgment of fact and estimation of frequency.

Problem

• LLMs' response to misinformation with uncertainty
• Impact of uncertainty on LLM fact-checking accuracy
• Correlation between fact judgment and frequency estimation

Innovation

• LLMs evaluate misinformation with uncertainty transformations.
• Analyze LLM responses to uncertain false statements.
• Correlate LLM fact judgments with frequency estimations.