ASTRA-QA: A Benchmark for Abstract Question Answering over Documents

2026-05-11

AI Summary

Existing document-based question answering benchmarks struggle to evaluate tasks requiring the synthesis of information from multiple sources to generate abstractive answers, primarily due to the absence of stable reference answers and fine-grained evaluation mechanisms. This work proposes ASTRA-QA, a new benchmark comprising 869 instances spanning five types of abstractive questions and three retrieval scopes. ASTRA-QA introduces, for the first time, explicit topic sets, annotations of unsupported content, and aligned evidence, enabling scalable and well-grounded automatic evaluation. Through topic coverage scoring and human-annotated hallucination detection, the benchmark demonstrates strong diagnostic capabilities across diverse retrieval-augmented generation (RAG) systems—including standard, graph-based, and hierarchical retrieval approaches—in terms of answer comprehensiveness, hallucination mitigation, and retrieval robustness, thereby establishing a reliable foundation for abstractive QA research.

Abstract

Document-based question answering (QA) increasingly includes abstract questions that require synthesizing scattered information from long documents or across multiple documents into coherent answers. However, this setting is still poorly supported by existing benchmarks and evaluation methods, which often lack stable abstract references or rely on coarse similarity metrics and unstable head-to-head comparisons. To alleviate this issue, we introduce ASTRA-QA, a benchmark for AbSTRAct Question Answering over documents. ASTRA-QA contains 869 QA instances over academic papers and news documents, covering five abstract question types and three controlled retrieval scopes. Each instance is equipped with explicit evaluation annotations, including answer topic sets, curated unsupported topics, and aligned evidence. Building on these annotations, ASTRA-QA assesses whether answers cover required key points and avoid unsupported content by directly scoring topic coverage and curated unsupported content, enabling scalable evaluation without exhaustive head-to-head comparisons. Experiments with representative Retrieval-Augmented Generation (RAG) methods spanning vanilla, graph-based, and hierarchical retrieval settings show that ASTRA-QA provides reference-grounded diagnostics for coverage, hallucination, and retrieval-scope robustness. Our dataset and code are available at https://xinyangsally.github.io/astra-benchmark.

Problem

• abstract question answering
• document-based QA
• evaluation benchmark
• topic coverage
• hallucination

Innovation

• Abstract Question Answering
• Benchmark
• Topic Coverage
• Hallucination Detection
• Retrieval-Augmented Generation