JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences

2025-08-22

AI Summary

This paper addresses the challenge of evaluating semantic preservation in French legal text simplification. We propose the first semantic consistency evaluation framework specifically designed for French legal sentence pairs. Our contributions are threefold: (1) We introduce FrJUDGE, the first annotated French legal semantic preservation dataset; (2) We develop JUDGEBERT—a domain-adapted BERT model incorporating legal linguistic features to compute semantic similarity, validated through controlled ablation studies; (3) In rigorous benchmarking—where identical sentence pairs receive 100% scores and unrelated pairs receive 0%—JUDGEBERT demonstrates robust performance, achieving high correlation with human judgments (Spearman’s ρ > 0.85) and significantly outperforming existing automated metrics. The framework enhances both accuracy and reliability in legal text simplification evaluation and shows strong potential for deployment in judicial NLP systems.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Simplifying text while preserving its meaning is a complex yet essential task, especially in sensitive domain applications like legal texts. When applied to a specialized field, like the legal domain, preservation differs significantly from its role in regular texts. This paper introduces FrJUDGE, a new dataset to assess legal meaning preservation between two legal texts. It also introduces JUDGEBERT, a novel evaluation metric designed to assess legal meaning preservation in French legal text simplification. JUDGEBERT demonstrates a superior correlation with human judgment compared to existing metrics. It also passes two crucial sanity checks, while other metrics did not: For two identical sentences, it always returns a score of 100%; on the other hand, it returns 0% for two unrelated sentences. Our findings highlight its potential to transform legal NLP applications, ensuring accuracy and accessibility for text simplification for legal practitioners and lay users.

Problem

• Assessing legal meaning preservation in text simplification
• Evaluating semantic equivalence between French legal sentences
• Developing specialized metrics for legal NLP applications

Innovation

• JUDGEBERT metric for legal meaning preservation
• FrJUDGE dataset assessing legal text pairs
• Superior human judgment correlation in evaluation