Legal Domain Adaptation of Modern BERT Models

2026-06-26

AI Summary

This work addresses the suboptimal performance of general-purpose large language models in the legal domain by proposing domain-adaptive continued pretraining of ModernBERT on the full corpus of U.S. court opinions. The approach integrates masked language modeling, long-sequence processing (supporting up to 8,192 tokens), and embedding generation with re-ranking techniques. Experimental results demonstrate that even though ModernBERT has been extensively pretrained on massive general-domain corpora, further pretraining on legal texts substantially enhances its performance on legal text understanding and retrieval tasks, outperforming training from scratch. The resulting model consistently and significantly surpasses the original ModernBERT across all evaluated datasets, efficiently producing high-quality legal paragraph embeddings and enabling rapid re-ranking of hundreds of legal passages.

Abstract

We investigate domain adaptation of modern BERT models in the legal domain. We further pre-train ModernBERT on all US court opinions using the masked language modeling objective. Although ModernBERT has been trained on roughly 500x more data than original BERT, we still find that this model benefits from further pre-training and domain adaptation in the legal domain: we report significant improvements compared to vanilla ModernBERT on all datasets connected to US court opinions. We find gains similar to those reported in early work on domain adaptation of BERT-like models. However, from scratch pre-training does not match the performance of further pre-training an existing ModernBERT checkpoint in our experiments. The resulting models are capable of processing sequences up to 8,192 tokens, and can be used to compute meaningful embeddings of legal passages, or could quickly rerank hundreds of legal passages for a given search query. We release all model checkpoints publicly.

Problem

• domain adaptation
• legal domain
• BERT models
• masked language modeling
• legal text processing

Innovation

• domain adaptation
• legal BERT
• further pre-training
• long-sequence modeling
• legal text embeddings