A comparison of pipelines for the translation of a low resource language based on transformers

2025-09-15

AI Summary

This study addresses machine translation for Bambara—a representative low-resource African language with approximately 14.2 million speakers—where data scarcity severely limits model performance. Method: We systematically compare three Transformer-based paradigms: (i) training from scratch, (ii) fine-tuning a large language model (LLaMA3), and (iii) student–teacher knowledge distillation leveraging LaBSE and BERT-enhanced architectural extensions. We propose a novel cross-lingual knowledge distillation framework tailored to low-resource settings to strengthen semantic representation learning. Experiments are conducted on the Bayelemagaba benchmark and a newly constructed Yiri dataset, evaluated using BLEU and chrF. Contribution/Results: The baseline Transformer achieves the best performance under low-resource constraints, attaining 33.81 BLEU and 41.0 chrF on Yiri. Simpler architectures consistently outperform complex fine-tuning approaches, highlighting a critical trade-off between model simplicity and data efficiency in low-resource MT.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This work compares three pipelines for training transformer-based neural networks to produce machine translators for Bambara, a Mandè language spoken in Africa by about 14,188,850 people. The first pipeline trains a simple transformer to translate sentences from French into Bambara. The second fine-tunes LLaMA3 (3B-8B) instructor models using decoder-only architectures for French-to-Bambara translation. Models from the first two pipelines were trained with different hyperparameter combinations to improve BLEU and chrF scores, evaluated on both test sentences and official Bambara benchmarks. The third pipeline uses language distillation with a student-teacher dual neural network to integrate Bambara into a pre-trained LaBSE model, which provides language-agnostic embeddings. A BERT extension is then applied to LaBSE to generate translations. All pipelines were tested on Dokotoro (medical) and Bayelemagaba (mixed domains). Results show that the first pipeline, although simpler, achieves the best translation accuracy (10% BLEU, 21% chrF on Bayelemagaba), consistent with low-resource translation results. On the Yiri dataset, created for this work, it achieves 33.81% BLEU and 41% chrF. Instructor-based models perform better on single datasets than on aggregated collections, suggesting they capture dataset-specific patterns more effectively.

Problem

• Comparing transformer pipelines for French-Bambara machine translation
• Evaluating low-resource translation methods using BLEU and chrF metrics
• Testing performance on medical and mixed-domain Bambara datasets

Innovation

• Transformer training for French-Bambara translation
• Fine-tuning LLaMA3 instructor models with decoder architectures
• Language distillation with student-teacher LaBSE integration