LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering

2026-07-02

AI Summary

This work addresses the scarcity of authentic spoken question-answering data for Luxembourgish—a low-resource language—which hinders the deployment of speech foundation models. The authors propose a parameter-efficient SLAM architecture that leverages multi-source text-to-speech (TTS) systems (MMS-TTS, Qwen3-TTS, and OmniVoice) to synthesize spoken Luxembourgish questions, which are then paired with textual QA data to form a training set. The model integrates a frozen Whisper encoder with a multilingual large language model via a learnable projection layer and LoRA adapters. Experiments demonstrate that the multi-source, task-oriented TTS synthesis strategy substantially outperforms single-source alternatives, yielding significant gains in spoken QA performance on the real-speaker test set LLAMA-LB-Test. Notably, TTS naturalness scores show no monotonic correlation with downstream task effectiveness, underscoring the need to evaluate synthetic speech quality based on task performance rather than perceptual metrics alone.

Abstract

Spoken Question Answering (SQA) remains largely focused on high-resource languages and carefully recorded speech, limiting the reach of speech-LLM methods in low-resource settings. This paper investigates whether text-to-speech (TTS) can provide task-specific training data for Luxembourgish SQA without requiring a large human-recorded QA corpus. Starting from existing text-based QA resources, we translate questions into Luxembourgish, synthesize spoken questions with multiple TTS systems, and pair them with textual answers. We train a parameter-efficient SLAM-style architecture that connects a frozen Whisper encoder to frozen multilingual LLM backends through a learned projector and LoRA adapters. We compare MMS-TTS, Qwen3-TTS, and OmniVoice variants, including single-source corpora of about 48k questions and a 4TTS multi-source mix of approximately 230k questions. Evaluation on LLAMA-LB-Test with two real Luxembourgish speaker conditions shows that multi-source and voice-design-based synthetic training configurations yield the strongest SQA performance. The results also show that no-reference TTS quality scores do not monotonically predict downstream QA performance, indicating that synthetic speech must be evaluated as task-specific training data rather than only as natural-sounding audio.

Problem

• Spoken Question Answering
• Low-resource languages
• Luxembourgish
• Text-to-Speech
• Synthetic speech

Innovation

• Spoken Question Answering
• Text-to-Speech Augmentation
• Low-resource Language
• Parameter-efficient Learning
• Synthetic Speech Evaluation