MathShikkha: A Controlled Study of Answer-Only and Chain-of-Thought Supervision for Bangla Mathematical Reasoning in Small Language Models

2026-08-09

AI Summary

This study addresses the lack of systematic investigation into the effectiveness of chain-of-thought (CoT) supervision for low-resource languages, focusing on Bengali. The authors introduce MathShikkha, the first Bengali mathematical reasoning dataset, with CoT annotations generated by GPT-4. Through rigorously controlled experiments comparing CoT supervision against answer-only supervision on 4B–7B parameter models, this work presents the first such comparative analysis in a low-resource language setting. Results demonstrate that CoT prompting yields substantial in-domain performance gains across all models (+20.1 to +28.1 accuracy points, p<0.0001) and improves out-of-domain robustness. In contrast, answer-only fine-tuning provides no significant benefit for stronger models and can even impair generalization. These findings underscore the critical role of CoT in ensuring linguistic consistency, auditability, and resilience under distributional shift.

Abstract

Mathematical reasoning remains challenging in low-resource languages such as Bangla. We study whether teacher-generated Bangla Chain-of-Thought (CoT) supervision provides benefits beyond ordinary supervised fine-tuning. We construct \textsc{MathShikkha}, a Bangla mathematical reasoning dataset with GPT-5.4-generated rationales, and fine-tune four 4B--7B student models under a matched protocol in which answer-only and CoT conditions share data splits, response-only loss masking, decoding, and scoring, differing only in the training target. In-domain, CoT provides no significant improvement over answer-only fine-tuning for three stronger backbones (paired bootstrap 95\% CIs include zero; exact McNemar $p \geq 0.17$), despite generating 15--52$\times$ more tokens, but significantly improves the weaker 4B model by 18.56 points ($p < 0.0001$). On the larger, contamination-audited BanglaMATH benchmark, this pattern reverses: CoT significantly outperforms answer-only supervision for all four models by 20.1--28.1 points (all $p < 0.0001$). Answer-only fine-tuning also reduces out-of-domain accuracy below the base model for three models, whereas CoT preserves or improves it for all four. A human study with two co-author annotators, external-expert adjudication, and Cohen's $κ= 0.76$--$1.00$ finds no significant CoT improvement over the base model on reasoning-content criteria; instead, its measurable effect is target-language adherence and producing inspectable reasoning. Overall, rationale supervision's value depends on backbone capability and distribution shift: in this setting, its main benefits are Bangla adherence, auditable reasoning, and out-of-domain robustness rather than improved in-domain reasoning validity.

Problem

• mathematical reasoning
• low-resource languages
• Chain-of-Thought
• Bangla
• small language models

Innovation

• Chain-of-Thought
• low-resource languages
• mathematical reasoning
• controlled fine-tuning
• out-of-domain robustness