Problem
Research questions and friction points this paper is trying to address.
Sinhala
Language Models
Text Classification
Sentiment Analysis
HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head
🤖 AI Summary
为提高僧伽罗语理解能力，通过从大量文本预训练两个基于BERT的模型，并提出双池分类头方法，在情感分析等任务上取得改进。
📝 Abstract
We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Dual Pooling Classification Head
Sinhala Language Understanding
BERT-based Model
🔎 Similar Papers
No similar papers found.
