Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification

2025-04-02

AI Summary

This paper addresses three critical bottlenecks hindering the real-world deployment of legal AI: scarcity of high-quality legal data, heavy reliance on domain experts for annotation, and low output reliability in high-stakes scenarios. To tackle these challenges, we propose a human-in-the-loop data governance framework grounded in a novel tripartite task paradigm—*data curation*, *expert-augmented collaborative annotation*, and *verifiability assessment*. Our methodology integrates legal knowledge graph construction, structured human-AI annotation protocols, and an output verification mechanism based on case-law consistency. We further design a systematic evaluation benchmark to quantify performance across dimensions of accuracy, auditability, and robustness. The framework has been validated in multiple judicial applications, demonstrating substantial improvements in annotation efficiency and result traceability. It provides a reusable technical pathway and foundational open-resource ecosystem for developing high-reliability legal AI tools.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The application of AI tools to the legal field feels natural: large legal document collections could be used with specialized AI to improve workflow efficiency for lawyers and ameliorate the"justice gap"for underserved clients. However, legal documents differ from the web-based text that underlies most AI systems. The challenges of legal AI are both specific to the legal domain, and confounded with the expectation of AI's high performance in high-stakes settings. We identify three areas of special relevance to practitioners: data curation, data annotation, and output verification. First, it is difficult to obtain usable legal texts. Legal collections are inconsistent, analog, and scattered for reasons technical, economic, and jurisdictional. AI tools can assist document curation efforts, but the lack of existing data also limits AI performance. Second, legal data annotation typically requires significant expertise to identify complex phenomena such as modes of judicial reasoning or controlling precedents. We describe case studies of AI systems that have been developed to improve the efficiency of human annotation in legal contexts and identify areas of underperformance. Finally, AI-supported work in the law is valuable only if results are verifiable and trustworthy. We describe both the abilities of AI systems to support evaluation of their outputs, as well as new approaches to systematic evaluation of computational systems in complex domains. We call on both legal and AI practitioners to collaborate across disciplines and to release open access materials to support the development of novel, high-performing, and reliable AI tools for legal applications.

Problem

• Legal documents differ from web-based text, requiring specialized AI approaches
• Data curation, annotation, and verification are key challenges in legal AI
• High expertise needed for legal data annotation and output verification

Innovation

• AI-assisted legal document curation
• Expert-guided legal data annotation
• Verifiable AI output evaluation