Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA

2026-01-24

AI Summary

Existing legal question-answering benchmarks predominantly focus on case law, overlooking the retrieval challenges and hallucination risks inherent in statutory law due to its hierarchical structure. This work proposes SearchFireSafety—the first statute-oriented, structure- and safety-aware QA benchmark—which jointly evaluates models’ ability to retrieve evidence within hierarchical regulations and their capacity to safely abstain from answering under insufficient information. We introduce graph-guided retrieval to model statutory structure and devise a hybrid data construction approach that integrates citation-aware retrieval with partial-context stress testing. Experiments demonstrate that graph-guided retrieval substantially improves performance; however, current large language models remain prone to severe hallucinations when context is incomplete, underscoring the necessity of co-evaluating structural awareness and safe refusal behaviors.

Abstract

Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning. In statutory domains, relevant evidence is distributed across hierarchically linked documents, creating a statutory retrieval gap where conventional retrievers fail and models often hallucinate under incomplete context. We introduce SearchFireSafety, a structure- and safety-aware benchmark for statute-centric legal QA. Instantiated on fire-safety regulations as a representative case, the benchmark evaluates whether models can retrieve hierarchically fragmented evidence and safely abstain when statutory context is insufficient. SearchFireSafety adopts a dual-source evaluation framework combining real-world questions that require citation-aware retrieval and synthetic partial-context scenarios that stress-test hallucination and refusal behavior. Experiments across multiple large language models show that graph-guided retrieval substantially improves performance, but also reveal a critical safety trade-off: domain-adapted models are more likely to hallucinate when key statutory evidence is missing. Our findings highlight the need for benchmarks that jointly evaluate hierarchical retrieval and model safety in statute-centric regulatory settings.

Problem

• statute-centric legal QA
• hierarchical retrieval
• hallucination
• model safety
• regulatory reasoning

Innovation

• structure-aware retrieval
• statute-centric QA
• graph-guided retrieval
• hallucination mitigation
• safety-aware benchmark