Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language

2026-06-15

AI Summary

This work addresses the high computational cost and deployment challenges of large language models for low-resource, non-Latin scripts such as Bengali on edge devices by introducing a compact, decoder-only base model with 135 million parameters specifically optimized for Bengali. The approach integrates the TituLLMs and SmolLM2-135M vocabularies and innovatively employs a deterministic intersection-based token merging strategy to mitigate subword fragmentation while preserving pretraining parameter stability. Evaluated on benchmarks including PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU, the model achieves performance comparable to that of models ranging from 270M to 1B parameters, substantially enhancing linguistic understanding capabilities in small-scale architectures. The code and model weights are publicly released.

Abstract

While the NLP landscape is dominated by multi-billion parameter architectures, their deployment in low-resource, non-Latin scripts remains computationally prohibitive for edge configurations, mobile systems, and decentralized local hardware. This paper presents bangla-smollm-135m, a highly compact 135-million parameter decoder-only foundational model engineered explicitly for high-efficiency language modeling in the Bangla script. By leveraging a deterministic intersect-and-append token merging strategy between TituLLMs and SmolLM2-135M, the model overcomes subword script fragmentation without destabilizing early pretrained parameter states. In zero-shot multi-task benchmark evaluations (PIQA_bn, OpenBookQA_bn, CommonsenseQA_bn, and Bangla_MMLU), bangla-smollm-135m matches or outperforms models twice its size (Gemma-3-270m) and achieves parity with models in the 1B parameter tier. The model is available at rnnandi/bangla-smollm-135m

Problem

• low-resource languages
• Bangla language
• efficient deployment
• non-Latin scripts
• edge computing

Innovation

• compact LLM
• token merging
• Bangla language
• zero-shot evaluation
• parameter efficiency