HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources

2026-07-03

AI Summary

Existing question answering systems lack a unified evaluation benchmark that supports joint retrieval from five heterogeneous sources: relational tables, text, images, spatial data, and knowledge graphs. This work proposes HETERQA, the first record-level QA benchmark encompassing all five modalities, comprising 857 answer-driven question-answer pairs grounded in Yelp business records. High-quality data are constructed through multi-source constraint initialization, heterogeneous information augmentation, and cross-source validation. Comprehensive experiments evaluating diverse retrieval and generation approaches reveal that hybrid retrieval achieves the highest Recall@10, while Self-RAG obtains the best MRR@10. However, overall performance remains far from saturated, underscoring the task’s inherent challenges and establishing a new direction for future research.

Abstract

In emerging systems (e.g., social media and e-commerce platforms), data records are often drawn from heterogeneous sources, such as relational tables, text documents, image repositories, spatial databases, and knowledge graphs. Accordingly, retrieving target records for question-answering (QA) tasks requires us to jointly exploit these heterogeneous sources. However, most existing benchmarks are constructed from individual sources, and only a very few recent benchmarks have considered two or three sources. To alleviate this issue, we introduce HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources. HETERQA instantiates this setting with Yelp business records, each of which is grounded by multiple sources. We build HETERQA in an answer-driven manner: candidate records are first initialized with record-field constraints, then enriched through heterogeneous sources, and finally cross-verified across required sources before the natural-language question is retained. We validate the benchmark through contradiction detection and human validation, and further evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers under the same metrics. The results show that HETERQA is challenging: hybrid retrieval achieves the strongest Recall@10, Self-RAG achieves the best MRR@10, and all evaluated methods remain far from saturating the benchmark. These findings indicate that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods. The benchmark dataset and source code are publicly available at https://huggingface.co/datasets/hanchang02/HeterQA and https://github.com/hanchang02/HeterQA, respectively.

Problem

• heterogeneous sources
• record retrieval
• question answering
• benchmark
• multi-source

Innovation

• heterogeneous retrieval
• record retrieval
• multi-source QA
• benchmark construction
• answer-driven validation