AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction

2026-01-30

AI Summary

Legal judgments are prone to errors due to case complexity and the abstract nature of legal norms, yet existing appellate mechanisms struggle to efficiently identify and correct post-hoc errors. This work introduces, for the first time, the “Appellate Review” task, which focuses on detecting, categorizing, and rectifying errors after a judgment has been issued. To support this endeavor, we construct AR-Bench, the first fine-grained benchmark dataset comprising 8,700 annotated judgments and 34,617 supplementary legal documents. Leveraging a structured error taxonomy and a large language model (LLM) evaluation framework, our empirical analysis reveals significant limitations of prevailing LLMs in identifying misapplications of law, thereby advancing legal AI from predictive generation toward reliability-oriented diagnosis.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts, while existing appellate review mechanisms face efficiency pressures from a surge in case volumes. Although current legal AI research focuses on tasks like judgment prediction and legal document generation, the task of judgment review differs fundamentally in its objectives and paradigm: it centers on detecting, classifying, and correcting errors after a judgment is issued, constituting anomaly detection rather than prediction or generation. To address this research gap, we introduce a novel task APPELLATE REVIEW, aiming to assess models'diagnostic reasoning and reliability in legal practice. We also construct a novel dataset benchmark AR-BENCH, which comprises 8,700 finely annotated decisions and 34,617 supplementary corpora. By evaluating 14 large language models, we reveal critical limitations in existing models'ability to identify legal application errors, providing empirical evidence for future improvements.

Problem

• legal reasoning
• judgment error
• appellate review
• anomaly detection
• legal AI

Innovation

• Appellate Review
• Legal Reasoning
• Judgment Error Detection
• AR-BENCH
• Anomaly Detection