🤖 AI Summary
This study addresses the challenge of automatically identifying negative citations in legal case law, a task highly susceptible to high-risk misclassifications that conventional accuracy metrics fail to adequately capture. To bridge this gap, the authors construct the first fine-grained, multi-label expert-annotated dataset specifically tailored to this task, comprising 239 real-world legal citations. They further propose “average severity error” as a more legally meaningful evaluation metric aligned with practical judicial concerns. A systematic benchmarking of prominent large language models—including Gemini 2.5 Flash and GPT-5-mini—reveals that Gemini 2.5 Flash achieves 79.1% accuracy in coarse-grained classification, while GPT-5-mini performs best on fine-grained tasks with 67.7% accuracy. These findings highlight nuanced differences in models’ capabilities for complex legal reasoning and establish a reliable baseline for future research in this domain.
📝 Abstract
Automating the classification of negative treatment in legal precedent is a critical yet nuanced NLP task where misclassification carries significant risk. To address the shortcomings of standard accuracy, this paper introduces a more robust evaluation framework. We benchmark modern Large Language Models on a new, expert-annotated dataset of 239 real-world legal citations and propose a novel Average Severity Error metric to better measure the practical impact of classification errors. Our experiments reveal a performance split. Google's Gemini 2.5 Flash achieved the highest accuracy on a high-level classification task (79.1%), while OpenAI's GPT-5-mini was the top performer on the more complex fine-grained schema (67.7%). This work establishes a crucial baseline, provides a new context-rich dataset, and introduces an evaluation metric tailored to the demands of this complex legal reasoning task.