XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

2026-09-06

AI Summary

研究通过构建XYBench评估大语言模型能否有效识别用户查询中的误解并提供实用解决方案，发现当前最强模型在这方面表现不佳。

Abstract

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g.,"How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that assesses model responses along three criteria grounded in cooperative response theory: (a) presence and (b) emphasis on pragmatic solutions, and (c) identification of misconceptions. Our experiments show that even the strongest LLMs predominantly answer the literal request (0.75--0.92) and far less often the intended one (0.33--0.71), while substantially lagging behind humans at identifying misconceptions (at most 63% vs. 79--90%). Further, models overwhelmingly prefer pragmatic responses in a multiple choice setting yet consistently fail to generate them. Oracle ablation experiments show that providing explicit user intent at generation time helps; however a large gap remains, suggesting pragmatic redirection is a fundamentally underdeveloped capability in current LLMs.

Problem

• misconceptions
• pragmatic solutions
• XY-problem

Innovation

• XYBench
• misconceptions in queries
• pragmatic solutions
• cooperative response theory
• large language models