Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

2026-06-20

AI Summary

This study addresses the limitations of existing medical encoder pretraining, which relies on small-scale, manually curated corpora that lack scalability and stylistic diversity—particularly acute in non-English clinical settings. To overcome this, the authors propose a web-scale pretraining approach tailored to French-language medical text. Their method first filters high-quality web pages based on medical term density, then enhances contextual richness by rewriting selected passages using a large language model. This work provides the first empirical validation of web-scale data for medical masked language modeling (MLM) pretraining, demonstrating that term-density filtering outperforms conventional educational-quality heuristics. The resulting DoctoBERT encoder, pretrained on the newly constructed FineMed corpus, achieves state-of-the-art performance on both the public DrBenchmark benchmark and proprietary clinical named entity recognition tasks.

Abstract

Web data curation has been widely studied for decoder Large Language Model (LLM) pretraining. Encoders for dense-terminology domains such as medicine, by contrast, are pretrained on small, manually-curated corpora that limit scalability and writing style diversity, a bottleneck even more severe in non-English clinical settings. Whether web-scale data curation also benefits encoder Masked Language Modeling (MLM) in a dense-terminology domain remains an open question. To address this, we introduce two complementary levers. Medical-term density filtering selects documents rich in medical terms. Signal-amplifying rephrasing uses an LLM to rewrite documents into denser variants with broader entity contexts. We instantiate the recipe on French medical NLP. The medical-term density filter outperforms the widely-used educational quality filter on downstream medical tasks, and the two complement each other. Signal-amplifying rephrasing alone improves on raw web data, and mixing it with filtered web data produces the largest gain. The recipe yields FineMed, a French medical pretraining corpus, and DoctoBERT, a state-of-the-art French medical encoder family evaluated on both the public benchmark DrBenchmark and a proprietary clinical Named Entity Recognition (NER) task.

Problem

• medical encoder pretraining
• web data curation
• dense-terminology domain
• Masked Language Modeling
• non-English clinical NLP

Innovation

• medical-term density filtering
• signal-amplifying rephrasing
• encoder pretraining
• web-scale data curation
• Masked Language Modeling