Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models

2024-09-17

arXiv.org

AI Summary

Existing audio-language models, though multilingually pretrained, exhibit strong English bias and poor instruction-following capabilities—particularly for low-resource languages like Thai—while lacking emergent cross-lingual generalization. To address this, we propose Typhoon-Audio: the first unified framework for audio understanding and spoken instruction following. It builds upon a multilingual audio encoder and employs progressive bilingual data mixing, integrating speech recognition, audio captioning, and structured spoken instruction data for instruction tuning. Crucially, this approach enhances Thai instruction adherence without degrading English performance. Experiments demonstrate that Typhoon-Audio significantly outperforms existing open-source models on Thai–English bilingual instruction tasks, matching Gemini-1.5-Pro’s performance. For the first time, our work systematically validates and alleviates the cross-lingual generalization bottleneck in multilingual audio models for low-resource languages.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.

Problem

• Improving audio models for low-resource languages like Thai
• Addressing lack of cross-lingual abilities in multilingual audio models
• Balancing language-specific and multilingual training for better performance

Innovation

• Optimizes data mixtures for target and English languages
• Integrates audio comprehension and speech instruction-following
• Balances language-specific and multilingual training data