BamiBERT: A New BERT-based Language Model for Vietnamese

2026-07-02

AI Summary

This work addresses key limitations of existing Vietnamese text encoders, particularly PhoBERT, which suffers from constrained context length and reliance on external tokenization. To overcome these issues, the authors train a BERT-based pretrained language model from scratch on 129 GB of general-domain Vietnamese text. The resulting model processes raw text directly without requiring external tokenizers and supports context lengths up to 2,048 tokens. Evaluated across eight Vietnamese benchmark tasks encompassing 15 metrics, it achieves state-of-the-art performance on 11 metrics and runner-up results on three, significantly outperforming prior approaches. This establishes the model as the strongest and most generalizable Vietnamese encoder currently available at the base scale.

Abstract

In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT

Problem

• Vietnamese language model
• context length limitation
• word segmentation
• text encoding
• pre-trained language model

Innovation

• BamiBERT
• Vietnamese language model
• extended context length
• word segmentation-free
• pre-trained BERT