SANA: What Matters for QA Agents over Massive Data Lakes?

2026-06-11

AI Summary

This work addresses the challenge of diagnosing failure sources in exploratory question-answering systems over data lakes, where tight coupling among search, planning, data analysis, and action policy modules obscures root causes. The authors propose SANA, a diagnostic ablation framework that constructs runtime profiles comprising gold source sequences, sanitized subproblems, and execution traces, enabling idealized module substitution and systematic ablation. For the first time, this approach facilitates interpretable diagnosis and independent evaluation of the four core components. Experiments on LakeQA and KramaBench reveal that data analysis constitutes the primary bottleneck, search imposes significant constraints in large-scale data lakes, and planning has comparatively minor impact. SANA enables systematic comparison and targeted improvements, establishing a standardized evaluation protocol for agent optimization.

Abstract

Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results. End-to-end accuracy alone cannot distinguish failures in search, planning, data analysis, or the agent's Action Policy: its decisions about what to do next and when to submit an answer. We present SANA (Search Agent Navigation Ablation framework), a diagnostic ablation framework that transforms EQA tasks into runtime profiles containing gold source sequence, sanitized subquestions, and execution records. SANA uses these profiles to construct idealized search, planning, and data-analysis tools, allowing each component to be ablated; the residual gap is diagnostic evidence for policy failures. To illustrate SANA as a reusable evaluation framework, we adapted two recent EQA benchmarks, LakeQA and KramaBench, and evaluated lightweight and mid-sized agents under fixed prompts, budgets, data lakes, and runtimes. Across both benchmarks, data analysis is a consistent bottleneck while planning is less so. Search is a major limitation in LakeQA's large data-lake setting, but less so for the smaller-scale KramaBench. SANA thus deconstructs end-to-end task accuracies into a diagnosis of where data-lake agents fail, and allows for systematic comparisons of progress in search, planning, data analysis, and agent design.

Problem

• Exploratory Question Answering
• Data Lakes
• LLM Agents
• Agent Evaluation
• Failure Diagnosis

Innovation

• SANA
• exploratory question answering
• data lakes
• ablation framework
• agent policy