Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis

2025-11-06

AI Summary

Natural language queries in table-based question answering (TQA) exhibit inherent ambiguity, yet existing research fails to systematically distinguish among three distinct query types—unambiguous, ambiguously cooperative, and non-cooperative—leading to biased evaluation. Method: This paper introduces the novel paradigm of “collaborative ambiguity,” reframing ambiguity as an interactive feature reflecting shared human-AI responsibility rather than a flaw, and proposes a query classification framework grounded in Grice’s Cooperative Principle. Through theoretical modeling and empirical analysis, we conduct systematic annotation and evaluation across 15 mainstream TQA benchmarks. Contribution/Results: Our study is the first to reveal pervasive query-type mixing across these datasets and demonstrates that current evaluation protocols conflate system execution accuracy with semantic understanding capability. The findings establish a theoretical foundation and practical methodology for developing interpretable, robust TQA interfaces and type-aware evaluation standards.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Intelligent Query Processing
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Natural language interfaces to tabular data must handle ambiguities inherent to queries. Instead of treating ambiguity as a deficiency, we reframe it as a feature of cooperative interaction where users are intentional about the degree to which they specify queries. We develop a principled framework based on a shared responsibility of query specification between user and system, distinguishing unambiguous and ambiguous cooperative queries, which systems can resolve through reasonable inference, from uncooperative queries that cannot be resolved. Applying the framework to evaluations for tabular question answering and analysis, we analyze the queries in 15 popular datasets, and observe an uncontrolled mixing of query types neither adequate for evaluating a system's execution accuracy nor for evaluating interpretation capabilities. This conceptualization around cooperation in resolving queries informs how to design and evaluate natural language interfaces for tabular data analysis, for which we distill concrete directions for future research and broader implications.

Problem

• Address ambiguity in natural language queries for tabular data analysis
• Develop cooperative framework distinguishing resolvable and unresolvable queries
• Evaluate query classification accuracy and interpretation in existing datasets

Innovation

• Framework distinguishes ambiguous cooperative queries
• System resolves ambiguity through reasonable inference
• Shared responsibility between user and system