Evaluating Reasoning Models for Queries with Presuppositions

2026-05-04

AI Summary

This study addresses the challenge of large language models in identifying and correcting factual inaccuracies embedded in user queries with erroneous presuppositions. It presents the first systematic evaluation of large reasoning models in such scenarios, introducing a novel dataset comprising queries with varying strengths of false presuppositions across health, scientific, and commonsense domains. The work compares the performance of large reasoning models against non-reasoning counterparts in both presupposition detection and response accuracy. Results reveal that reasoning models offer only marginal improvements (2–11% higher accuracy) and still fail to challenge 26–42% of incorrect presuppositions. Furthermore, their performance is significantly influenced by the linguistic strength of the presupposition, highlighting a critical limitation in their ability to robustly contest flawed premises.

Abstract

Millions of users turn to AI models for their information needs. It is conceivable that a large number of user queries contain assumptions that may be factually inaccurate. Prior work notes that large language models (LLMs) often fail to challenge such erroneous assumptions, and can reinforce users' misinformed opinions. However, given the recent advances, especially in model's reasoning capabilities, we revisit whether large reasoning models (LRMs) can reason about the underlying assumptions and respond to user queries appropriately. We construct queries with varying degrees of presuppositions spanning health, science, and general knowledge, and use it to evaluate several widely-deployed models When compared to non-reasoning models, we find that reasoning models achieve a slightly higher accuracy (2-11%), but they still fail to challenge a large fraction (26-42%) of false presuppositions. Further, reasoning models remain susceptible to how strongly the presupposition is expressed.

Problem

• presuppositions
• reasoning models
• large language models
• factuality
• misinformation

Innovation

• reasoning models
• presupposition
• fact-checking
• large language models
• critical reasoning