LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes

2026-07-13

AI Summary

Existing question-answering systems lack an end-to-end evaluation benchmark for real-world data lakes comprising heterogeneous, weakly structured tables, text, and metadata. This work introduces a novel benchmark spanning three domains—AI/ML metadata, retail banking, and multimodal biomedical data—that features human-validated question-answer pairs with precise, modality-aware evidence pointers. The benchmark explicitly decouples source discovery from cross-modal reasoning and supports the evaluation of diverse approaches, including retrieval-augmented generation (RAG) and agent-based tool invocation. Empirical results reveal that even state-of-the-art systems exhibit significant shortcomings in tasks requiring relational chain reasoning, policy grounding, and cross-table compositional question answering, despite high-quality retrieval performance.

Abstract

While modern question answering (QA) systems excel on clean, schema-aligned corpora, real-world knowledge is rarely so neatly packaged. Answering questions over enterprise and scientific data lakes requires systems to navigate heterogeneous, weakly structured collections of tables, passages, and linked metadata. Current benchmarks abstract away this noisy discovery process, failing to evaluate end-to-end performance. To bridge this gap, we introduce LakeQuest, a human-validated benchmark of 9,846 QA pairs designed to evaluate the end-to-end retrieve-and-synthesize pipeline over realistic data lakes. LakeQuest spans three diverse domains (AI/ML metadata, retail banking, and multimodal biomedical drug information) and pairs every question with exact, modality-aware evidence pointers. By isolating source discovery from cross-modal synthesis, LakeQuest exposes critical failure modes in modern QA systems. Our baseline evaluations, including standard Retrieval-Augmented Generation (RAG) and agentic tool-use methods, reveal that high-quality retrieval does not guarantee correct reasoning. Systems consistently struggle with relation chaining in metadata graphs, policy grounding in bank ledgers, and joint tabular QA in biomedical contexts, highlighting the need for robust discovery and faithful cross-file composition mechanisms in future agentic QA systems.

Problem

• data lakes
• question answering
• heterogeneous data
• source discovery
• cross-modal synthesis

Innovation

• data lakes
• grounded question answering
• retrieve-and-synthesize pipeline
• modality-aware evidence
• cross-modal synthesis