Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?

2024-09-27

arXiv.org

AI Summary

This work investigates whether large language models (LLMs) can perform extreme low-resource (XLR) machine translation using only a grammar book, and analyzes the underlying mechanisms. Methodologically, the authors systematically ablate components of grammar books—parallel example sentences versus grammatical rule explanations—and fine-tune encoder-decoder models on such data. Results show that parallel examples—not rule descriptions—are the primary driver of translation performance; fine-tuned encoder-decoder models match or exceed long-context LLM prompting. Key contributions include: (i) the first empirical demonstration that parallel examples in grammar books are critical for XLR translation; (ii) a typologically informed prompting framework for linguistic tasks; and (iii) the “task-adapted data” principle—translation requires parallel corpora, while linguistic analysis relies on grammatical knowledge. Experiments on Kalamang, Nepali, and Guarani validate efficacy; typological prompting achieves SOTA on grammaticality judgment and lexical semantic labeling; and LLMs’ purported utilization of syntactic explanations is empirically refuted.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Extremely low-resource (XLR) languages lack substantial corpora for training NLP models, motivating the use of all available resources such as dictionaries and grammar books. Machine Translation from One Book (Tanzer et al., 2024) suggests that prompting long-context LLMs with one grammar book enables English-Kalamang translation, an XLR language unseen by LLMs - a noteworthy case of linguistics helping an NLP task. We investigate the source of this translation ability, finding almost all improvements stem from the book's parallel examples rather than its grammatical explanations. We find similar results for Nepali and Guarani, seen low-resource languages, and we achieve performance comparable to an LLM with a grammar book by simply fine-tuning an encoder-decoder translation model. We then investigate where grammar books help by testing two linguistic tasks, grammaticality judgment and gloss prediction, and we explore what kind of grammatical knowledge helps by introducing a typological feature prompt that achieves leading results on these more relevant tasks. We thus emphasise the importance of task-appropriate data for XLR languages: parallel examples for translation, and grammatical data for linguistic tasks. As we find no evidence that long-context LLMs can make effective use of grammatical explanations for XLR translation, we conclude data collection for multilingual XLR tasks such as translation is best focused on parallel data over linguistic description.

Problem

• Investigating LLMs' ability to translate XLR languages using grammar books
• Assessing the impact of parallel examples versus grammatical explanations in translation
• Exploring task-appropriate data needs for XLR languages in NLP tasks

Innovation

• Using grammar books to prompt LLMs for translation
• Fine-tuning encoder-decoder models with parallel examples
• Typological feature prompts for linguistic tasks