NeoBERT: A Next-Generation BERT

2025-02-26

AI Summary

Existing bidirectional encoders—such as BERT and RoBERTa—lag behind modern autoregressive LLMs (e.g., LLaMA, DeepSeek) in architectural innovation, pretraining paradigms, and contextual capacity. To address this gap, we propose NeoBERT: a next-generation, efficient bidirectional encoder tailored to contemporary NLP demands. Its core innovations include: (1) the first unified integration of an empirically optimal depth-to-width ratio, 4096-token context modeling, modern pretraining corpora, and a compact 250M-parameter scale; (2) an enhanced Transformer architecture; and (3) evaluation under a unified MTEB/GLUE fine-tuning benchmark. Experiments demonstrate that NeoBERT achieves state-of-the-art performance on the MTEB benchmark, outperforming BERT-large, RoBERTa-large, and other leading encoders under identical fine-tuning protocols. All code, data, and model checkpoints are publicly released.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent innovations in architecture, pre-training, and fine-tuning have led to the remarkable in-context learning and reasoning abilities of large auto-regressive language models such as LLaMA and DeepSeek. In contrast, encoders like BERT and RoBERTa have not seen the same level of progress despite being foundational for many downstream NLP applications. To bridge this gap, we introduce NeoBERT, a next-generation encoder that redefines the capabilities of bidirectional models by integrating state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. NeoBERT is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions. In addition, we rigorously evaluate the impact of each modification on GLUE and design a uniform fine-tuning and evaluation framework for MTEB. We release all code, data, checkpoints, and training scripts to accelerate research and real-world adoption.

Problem

• Enhance bidirectional encoder performance
• Integrate modern architecture and data
• Optimize pre-training for NLP applications

Innovation

• Integrated advanced architecture enhancements
• Optimized pre-training methodologies applied
• Extended context length implementation