DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

2025-11-03

AI Summary

Existing open-domain QA benchmarks inadequately evaluate large language models’ (LLMs) answer completeness under intertwined name ambiguity and multi-step reasoning. Method: We propose DeepAmbigQAGen—the first automated data generation framework that jointly leverages textual corpora and linked knowledge graphs to explicitly model ambiguity and construct verifiable multi-hop questions. Contribution/Results: Using this framework, we build DeepAmbigQA, a challenging benchmark of 3,600 real-world, complex, and answer-verifiable ambiguous multi-hop questions. Experiments reveal severe limitations in current LLMs: even GPT-5 achieves only 0.13 exact-match accuracy, exposing fundamental deficiencies in answer completeness. DeepAmbigQA fills a critical gap in ambiguity-aware multi-hop QA evaluation, providing both a rigorous new benchmark and a methodological foundation for advancing model robustness and answer completeness research.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as Which actor from the film Heat won at least one Academy Award?, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.

Problem

• Benchmarking LLM completeness on ambiguous multi-hop questions
• Addressing name ambiguity and multi-step reasoning in QA systems
• Evaluating answer completeness for complex information gathering tasks

Innovation

• Generates questions with name ambiguity and multi-hop reasoning
• Constructs QA tasks from text corpora and knowledge graphs
• Automatically creates verifiable benchmark dataset for LLMs