Speechless: Speech Instruction Training Without Speech for Low Resource Languages

2025-05-23

AI Summary

Low-resource languages face a dual bottleneck in speech command understanding: scarcity of high-quality spoken command data and limited availability of suitable text-to-speech (TTS) models. Method: This paper proposes a novel, TTS-free paradigm for speech command understanding. It aligns Whisper encoder embeddings with synthetically generated text commands in semantic space, enabling large language models to comprehend speech inputs via pure text-based instruction tuning—without synthesizing speech. The approach comprises three components: cross-modal semantic alignment using the Whisper encoder, text-instruction-supervised fine-tuning, and semantic-space knowledge distillation—forming a zero-TTS training framework. Contribution/Results: Experiments demonstrate substantial improvements in speech command understanding accuracy for low-resource languages—even without any real spoken command data. At inference, the model processes raw audio directly. Training efficiency increases by over 3× compared to TTS-dependent baselines. To our knowledge, this is the first work to bypass TTS entirely and achieve speech–text command alignment solely at the semantic level.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid growth of voice assistants powered by large language models (LLM) has highlighted a need for speech instruction data to train these systems. Despite the abundance of speech recognition data, there is a notable scarcity of speech instruction data, which is essential for fine-tuning models to understand and execute spoken commands. Generating high-quality synthetic speech requires a good text-to-speech (TTS) model, which may not be available to low resource languages. Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS. We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference. This simplified training process is a promising approach to building voice assistant for low-resource languages.

Problem

• Addressing scarcity of speech instruction data for low-resource languages
• Eliminating need for text-to-speech models in speech instruction training
• Enabling LLMs to understand spoken commands without speech data

Innovation

• Halts synthesis at semantic representation level
• Aligns synthetic semantics with Whisper encoder
• Fine-tunes LLM on text instructions only