LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases

2025-12-14

AI Summary

Chinese civil case legal relation extraction has long been hindered by the absence of a systematic framework. Method: This paper introduces LexRel—the first benchmark dataset for this domain—along with a hierarchical legal relation taxonomy and fine-grained argument definitions to enable structured modeling of civil legal relations. Leveraging legal knowledge guidance, expert-collaborative annotation, and large language model (LLM) evaluation, we establish the first systematic pattern for Chinese civil legal relations. Contribution/Results: Empirical analysis reveals that state-of-the-art LLMs achieve less than 60% accuracy in legal relation identification, exposing critical deficiencies in fine-grained legal semantic understanding. Furthermore, integrating legal relation features improves downstream tasks—e.g., analogous case recommendation—by 3.2–5.8 percentage points, demonstrating their generalizability and practical utility.

Technology Category

• Machine Learning: Statistical Relational/Logic Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Legal relations form a highly consequential analytical framework of civil law system, serving as a crucial foundation for resolving disputes and realizing values of the rule of law in judicial practice. However, legal relations in Chinese civil cases remain underexplored in the field of legal artificial intelligence (legal AI), largely due to the absence of comprehensive schemas. In this work, we firstly introduce a comprehensive schema, which contains a hierarchical taxonomy and definitions of arguments, for AI systems to capture legal relations in Chinese civil cases. Based on this schema, we then formulate legal relation extraction task and present LexRel, an expert-annotated benchmark for legal relation extraction in Chinese civil law. We use LexRel to evaluate state-of-the-art large language models (LLMs) on legal relation extractions, showing that current LLMs exhibit significant limitations in accurately identifying civil legal relations. Furthermore, we demonstrate that incorporating legal relations information leads to consistent performance gains on other downstream legal AI tasks.

Problem

• Develops a schema for extracting legal relations in Chinese civil cases
• Creates a benchmark to evaluate AI models on legal relation extraction
• Assesses limitations of large language models in identifying civil legal relations

Innovation

• Introduces comprehensive hierarchical schema for legal relations
• Creates expert-annotated benchmark LexRel for Chinese civil cases
• Evaluates LLMs and integrates legal relations into downstream tasks