GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

2026-07-14

AI Summary

This study addresses the lack of effective evaluation benchmarks for mathematical reasoning in low-resource languages by introducing GSM-Plus-BN, the first perturbation-based math word problem dataset for Bengali. The dataset comprises 9,000 human-translated and verified questions augmented with semantic perturbations to enhance evaluation robustness. The authors conduct a systematic assessment of six open-source large language models using both standard and chain-of-thought prompting strategies. Results show that GPT-OSS-20B achieves 96.08% accuracy on original problems, with chain-of-thought prompting substantially improving performance. While larger models demonstrate greater robustness on perturbed instances, their overall performance still lags significantly behind English-language benchmarks. This work fills a critical gap in evaluating mathematical reasoning capabilities for under-resourced languages.

Abstract

The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali. Despite this global significance, there has been minimal prior work on mathematical reasoning in Bengali and no existing research that systematically benchmarks a perturbated Bengali mathematical dataset, leaving a critical void in assessing model robustness and true comprehension beyond pattern recognition. This study addresses this gap by introducing GSM-Plus-BN, a novel perturbated Bengali mathematical dataset derived from the English GSM-Plus benchmark and verified by human translators. We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples comprising 1,000 seed questions and 8,000 perturbed variants under both Standard Prompting and Chain-of-Thought (CoT) Prompting. Experimental results show that GPT-OSS-20B achieves the highest seed question accuracy of 96.08% under Standard Prompting, while larger models such as Llama-3.3-70B and GPT-OSS-120B demonstrate superior robustness across perturbation types. Furthermore, CoT prompting substantially improves reasoning for most models compared to Standard Prompting, yet a notable performance gap persists across all models relative to their English benchmarks, underscoring the inherent difficulty of perturbed Bengali text. This research makes a foundational contribution by providing GSM-PLUS-BN as a new resource and baseline for future Bengali mathematical reasoning research.

Problem

• mathematical reasoning
• Bengali
• large language models
• perturbation
• benchmark

Innovation

• perturbation-based benchmark
• Bengali mathematical reasoning
• low-resource language evaluation
• Chain-of-Thought prompting
• multilingual LLM robustness