Are you speaking my languages? On spoken language adherence in multimodal LLMs

2026-06-15

AI Summary

This work addresses the issue of language inconsistency in multimodal large language models for automatic speech recognition (ASR), where mismatches between the input speech language and model output lead to transcription errors and degraded downstream performance. The study formally defines this “language inconsistency” problem for the first time and introduces a soft prompting mechanism that guides the model to adhere to the input language without imposing hard constraints, thereby preserving its code-switching capability. A novel metric is proposed to quantitatively measure language violations. Evaluations across zero-shot prompting, supervised fine-tuning (SFT), and chain-of-thought (CoT) reasoning strategies demonstrate substantial reductions in language violation rates in multilingual ASR settings while maintaining overall recognition accuracy, offering practical deployment solutions adaptable to varying computational budgets.

Abstract

While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.

Problem

• language adherence
• multimodal LLMs
• Automatic Speech Recognition
• code-switching
• language identification

Innovation

• language adherence
• soft prompting
• multilingual ASR
• Chain-of-Thought reasoning
• supervised fine-tuning