🤖 AI Summary
This work addresses the challenge of efficiently adapting general-purpose speech-language models to Singapore’s multilingual, sociolinguistically sensitive contexts—such as those encountered by the Home Team—when access to original training data is unavailable. The authors propose an integrated approach combining LoRA-based fine-tuning, a novel multilingual question-answering dataset (HTD-multilingual-QA) designed to mitigate catastrophic forgetting, and an enhanced CoBa reweighting strategy extended for the first time to speech-based multitask learning. The resulting 5B-parameter model, HT-Moonstone, matches or surpasses the performance of models seven times its size across five multilingual speech tasks, achieving substantial gains in accent and gender recognition while incurring less than a 2% degradation in original speech question-answering capability.
📝 Abstract
Spoken language models (SLMs) unify speech perception and reasoning, but adapting them to sensitive domains is underexplored, especially when the original training data is inaccessible and the use case demands multilingual, spoken-query interaction. We adapt an open-source SLM to the Singaporean Home Team context across five speech tasks in Singapore's four official languages, combining LoRA fine-tuning, a surrogate text-QA dataset that guards against catastrophic forgetting, and a multi-task objective that adapts the CoBa reweighting scheme to speech. We also build HTD-multilingual-QA, a 504,853 sample multilingual QA dataset in text and spoken form. The resulting HT-Moonstone (5B) matches or outperforms SLMs up to 7x its size on most tasks, attains the best accent and gender recognition among all models evaluated, and loses under 2\% of its original speech QA ability.