🤖 AI Summary
This study addresses the challenge of high token consumption and performance degradation in transactional legal document question answering when naively injecting entire corpora into large language models. The authors propose a structure-aware chunking strategy and comparatively evaluate three approaches: full corpus injection, embedding-based retrieval (NAVEMBED), and a novel large model navigation framework leveraging a compact structural index (NAVINDEX). NAVINDEX introduces an innovative navigation mechanism coupled with a cached intersection cost model, achieving substantial reductions in token usage—1.61× fewer total tokens and a 56× smaller context window—alongside a 25% cost reduction, while maintaining answer quality on par with full injection. Experimental results show that NAVEMBED matches full injection on 16 out of 18 tasks with 17.3× fewer tokens, whereas NAVINDEX consistently matches performance across all tasks, demonstrating superior efficiency and cost-effectiveness.
📝 Abstract
Answering questions over a set of transactional legal documents is most simply done by injecting the whole corpus into the LLM's context window on every query. That baseline maximises retrieval recall, but its token footprint scales with the corpus rather than the question, and long-context degradation scales with it. We report what it took to replace full-corpus injection in a legal-document analysis system, comparing it against two structured retrieval modes over our proprietary structure-aware chunking: embedding retrieval (NAVEMBED) and LLM navigation over a compact structured index (NAVINDEX). On a 20-question benchmark with verified ground-truth answers, a position-bias-controlled, reference-anchored pairwise judge scored semantic retrieval with reranking tied with injection on 16 of 18 document-bound questions (injection preferred on 2) while attending to 17.3x fewer input tokens (a general-text-embedding (GTE) configuration reaches 29.9x at a lower tie rate); both modes were judged tied on the 2 out-of-scope controls. NAVINDEX was judged tied on all 18 at a 1.61x smaller total token footprint, a ~56x smaller answering context, and 25% lower dollar cost. We derive a closed-form caching-crossover rule: cached injection is cheaper in dollars only while the corpus stays below roughly ten times the retrieval payload. Scope and uncertainty are quantified in Section 8.