Is Complex Query Answering Really Complex?

2024-10-16

arXiv.org

AI Summary

Existing CQA benchmarks suffer from severe simplification bias: up to 98% of queries degenerate into single-hop link prediction, leading to inflated estimates of multi-hop reasoning capability. To address this, we introduce the first CQA benchmark explicitly designed for *irreducible, multi-hop-required* queries. Our method systematically analyzes query reducibility and imposes strict structural constraints to eliminate all simplification paths, while adopting a multi-hop-dominant evaluation protocol. On this rigorous benchmark, state-of-the-art CQA models exhibit substantial performance degradation—averaging over 40% drop—exposing critical deficiencies in generalization and deep logical reasoning. This work not only uncovers fundamental flaws in current evaluation paradigms but also establishes a more realistic, trustworthy assessment standard aligned with the intrinsic complexity of real-world knowledge graphs, thereby steering CQA research toward genuine multi-hop reasoning competence.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Question Answering
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Community question answering

Abstract

Complex query answering (CQA) on knowledge graphs (KGs) is gaining momentum as a challenging reasoning task. In this paper, we show that the current benchmarks for CQA might not be as complex as we think, as the way they are built distorts our perception of progress in this field. For example, we find that in these benchmarks, most queries (up to 98% for some query types) can be reduced to simpler problems, e.g., link prediction, where only one link needs to be predicted. The performance of state-of-the-art CQA models decreases significantly when such models are evaluated on queries that cannot be reduced to easier types. Thus, we propose a set of more challenging benchmarks composed of queries that require models to reason over multiple hops and better reflect the construction of real-world KGs. In a systematic empirical investigation, the new benchmarks show that current methods leave much to be desired from current CQA methods.

Problem

• Challenges in complex query answering benchmarks
• Reduction of complex queries to simpler problems
• Need for more realistic knowledge graph benchmarks

Innovation

• Introduces challenging CQA benchmarks
• Focuses on multi-hop reasoning
• Evaluates unreduced complex queries