Problem
Research questions and friction points this paper is trying to address.
Arabic NLP
masked language modeling
state-space models
long-sequence efficiency
bidirectional encoder
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
🤖 AI Summary
This work addresses the inefficiency of existing Arabic pre-trained models in processing long texts due to the quadratic complexity of self-attention mechanisms. To overcome this limitation, we propose the first bidirectional Mamba encoder tailored for Arabic, leveraging a selective state space architecture that enables linear-complexity modeling of long sequences. The model is pretrained from scratch via masked language modeling on Arabic Wikipedia and CulturaX corpora. Experimental results demonstrate that our approach achieves highly efficient training on consumer-grade GPUs and delivers strong performance across multiple benchmarks, including 96.37% accuracy on the HARD sentiment classification task, as well as competitive results on ARCD question answering and ANERcorp named entity recognition. Although it slightly underperforms Transformer baselines on XNLI inference, the model overall matches or surpasses current state-of-the-art methods in Arabic NLP.
📝 Abstract
Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT have become the standard backbone for Arabic natural language understanding, but their self-attention mechanism scales quadratically with sequence length, which limits efficiency on long documents. Mamba, a selective state-space model (SSM), offers linear-time sequence modeling as a competitive alternative to attention, yet no dedicated bidirectional Mamba encoder pretrained specifically for Arabic currently exists. We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days. We evaluate AraSSM by fine-tuning on four established Arabic NLU benchmarks covering sentiment classification (HARD), named entity recognition (ANERcorp), extractive question answering (ARCD), and natural language inference (XNLI-ar), following the per-task evaluation protocol introduced by AraBERT, and report results as mean +/- standard deviation across three fine-tuning seeds. AraSSM matches or exceeds published base-sized Transformer baselines on sentiment classification (96.37 +/- 0.03% accuracy on HARD), is competitive on extractive QA (32.19 +/- 1.07 EM, 63.79 +/- 0.25 F1 on ARCD) and named entity recognition (81.54 +/- 0.30 entity-level F1 on ANERcorp), and trails the base-sized Transformer range on natural language inference (72.83 +/- 0.07% accuracy on XNLI-ar), despite being trained entirely from scratch on consumer hardware rather than large-scale accelerator clusters.
Innovation
Methods, ideas, or system contributions that make the work stand out.
state-space model
bidirectional Mamba
masked language modeling
Arabic NLU
efficient sequence modeling
🔎 Similar Papers
No similar papers found.
