IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources

2026-06-18

AI Summary

This study addresses the limitations of Persian pre-trained language models, which suffer from scarce high-quality corpora and a narrow range of evaluation benchmarks. To overcome these challenges, the authors train from scratch a monolingual model based on the RoBERTa-base architecture. They introduce an innovative semantic deduplication method leveraging a vector database to ensure domain-balanced pretraining and develop a 139k BPE tokenizer tailored to Persian morphology and orthography. A multi-stage corpus cleaning pipeline—including normalization, anonymization, and near-duplicate removal—is employed to enhance data quality. The resulting model achieves strong performance, attaining F1 scores of 88.35 and 49.10 on the PQuAD and ParsiNLU-RC question answering tasks, respectively, and a Macro-F1 of 0.8350 on the FarsTail textual inference benchmark, outperforming or competitively matching existing approaches across multiple metrics.

Abstract

Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora and by insufficient evaluation beyond standard classification and NER tasks. We present IHUBERT, a monolingual Persian PLM trained from scratch with the RoBERTa-base encoder (125M parameters) on a 45 GB curated subset of the Sepahr-Danesh collection (about 7-8B tokens). To improve corpus quality and reduce redundancy, we employ a multi-stage preprocessing pipeline that includes normalization, exact and near-duplicate removal, anonymization, and vector-database-based semantic deduplication for distribution balancing control across domains and registers. We additionally train a 139k-vocabulary BPE tokenizer on the full pretraining corpus to better capture Persian morphology and orthographic variation. IHUBERT is evaluated on seven Persian NLU benchmarks covering NER, sentiment analysis, topic classification, NLI, extractive question answering, and relation extraction, using task-standard metrics (entity-level F1, Macro-F1, EM/F1). IHUBERT achieves its strongest gains on extractive QA, ranking first on both PQuAD (F1 88.3542) and ParsiNLU-RC (F1 49.0987), and attains the best result on FarsTail (Macro-F1 0.8350). On NER and topic classification, it remains competitive (e.g., 0.8308 F1 on ParsTwiNER; 0.7953 Macro-F1 on DigiMag), while relation extraction remains the main remaining gap (0.6684 Macro-F1 on PERLEX). A controlled tokenizer ablation on the IHUBERT pretraining corpus shows that BPE yields slightly lower subword fragmentation than WordPiece at matched vocabulary size, supporting our tokenization design. Overall, IHUBERT advances Persian language modeling through semantically curated large-scale pretraining and broad evaluation across both classification and comprehension-oriented tasks.

Problem

• Persian pretrained language models
• pretraining corpora scarcity
• evaluation limitation
• semantic redundancy
• domain imbalance

Innovation

• semantic deduplication
• domain-balanced pretraining
• Persian BPE tokenizer
• monolingual PLM
• vector-based corpus curation