LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents

2025-09-02

AI Summary

To address the low reliability and poor interpretability of large language models (LLMs) in long-text legal information retrieval, this paper proposes a structured prompting framework. It systematically partitions and augments long legal documents (e.g., CUAD) via chunking and data augmentation, then introduces an engineered prompt template incorporating a distribution-aware localization mechanism and an inverse-cardinality-weighted heuristic candidate filtering strategy—significantly mitigating black-box behavior. Implemented on the Qwen-2 architecture, the method achieves state-of-the-art performance on the CUAD benchmark, outperforming prior best approaches by 9%. Empirical analysis further exposes critical limitations of mainstream automated evaluation metrics in legal-domain tasks. This work establishes a novel paradigm for enhancing interpretability, accountability, and practical reliability of legal AI systems.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rise of Large Language Models (LLMs) has had a profoundly transformative effect on a number of fields and domains. However, their uptake in Law has proven more challenging due to the important issues of reliability and transparency. In this study, we present a structured prompting methodology as a viable alternative to the often expensive fine-tuning, with the capability of tacking long legal documents from the CUAD dataset on the task of information retrieval. Each document is first split into chunks via a system of chunking and augmentation, addressing the long document problem. Then, alongside an engineered prompt, the input is fed into QWEN-2 to produce a set of answers for each question. Finally, we tackle the resulting candidate selection problem with the introduction of the Distribution-based Localisation and Inverse Cardinality Weighting heuristics. This approach leverages a general purpose model to promote long term scalability, prompt engineering to increase reliability and the two heuristic strategies to reduce the impact of the black box effect. Whilst our model performs up to 9% better than the previously presented method, reaching state-of-the-art performance, it also highlights the limiting factor of current automatic evaluation metrics for question answering, serving as a call to action for future research. However, the chief aim of this work is to underscore the potential of structured prompt engineering as a useful, yet under-explored, tool in ensuring accountability and responsibility of AI in the legal domain, and beyond.

Problem

• Addressing reliability and transparency challenges of LLMs in legal domain
• Overcoming long document limitations for legal information retrieval
• Reducing black box effect through structured prompting heuristics

Innovation

• Structured prompting methodology for legal documents
• Chunking and augmentation for long documents
• Heuristics for candidate selection and weighting