Retrieval-Based Multi-Label Legal Annotation: Extensible, Data-Efficient and Hallucination-Free

2026-05-15

AI Summary

This work addresses the challenges of multi-label legal document annotation, including a vast and dynamically evolving label space, scarce supervised data, and the propensity of models to generate hallucinated labels. The authors propose a retrieval-based annotation framework that reformulates the task as semantic matching between documents and label descriptions. By leveraging frozen large-scale retrieval models—such as Qwen-8B—to produce embeddings and applying k-nearest neighbor classification for label prediction, the approach adapts seamlessly to label changes without fine-tuning. Evaluated on the Eurlex and ECtHR datasets, the method substantially outperforms zero-shot GPT and Legal-BERT, achieving nearly double the Micro-F1 score with only 100 training samples, reducing computational costs by 20–30×, and inherently eliminating label hallucination while demonstrating strong scalability and data efficiency.

Abstract

Multi-label legal annotation requires assigning multiple labels from large, evolving taxonomies to long, fact-intensive documents, often under limited supervision. Parametric encoders typically require task-specific training and retraining when the label set changes, while prompting generative large language models becomes costly and degrades as the label space grows. We cast legal annotation as retrieval: we embed documents and label descriptions with a frozen retrieval model and predict labels via k-nearest neighbors in the embedding space, enabling updates by re-embedding and re-indexing rather than gradient-based backpropagation. Across three legal datasets (ECtHR-A, ECtHR-B, and Eurlex with 100 labels), retrieval achieves competitive accuracy and strong data efficiency; on Eurlex, Qwen-8B retrieval improves Macro-F1 from 40.41 (GPT-5.2, zero-shot) to 49.12 while reducing estimated compute by 20-30 times compared to fine-tuning. With only (N=100) training samples, retrieval nearly doubles Micro-F1 over hierarchical Legal-BERT on ECtHR-A (48.29 vs. 27.87). We also quantify a reliability failure mode of generative inference: GPT-5.2 hallucinates labels outside the provided taxonomy in 0.12-0.9% of test samples under deterministic decoding. In contrast, retrieval strictly respects defined label sets, eliminating hallucination by design. These results suggest retrieval-model-based annotators are a practical, deployable alternative for high-cardinality and rapidly changing legal label spaces.

Problem

• multi-label legal annotation
• large evolving taxonomies
• limited supervision
• label hallucination
• high-cardinality label spaces

Innovation

• retrieval-based annotation
• multi-label legal classification
• hallucination-free
• data-efficient learning
• frozen embedding model