Text-only adaptation in LLM-based ASR through text denoising

2026-01-28

AI Summary

This work addresses the performance degradation commonly observed in large language model (LLM)-based speech recognition systems when adapting solely with in-domain text, a process that often disrupts the alignment between speech and text modalities. To mitigate this issue, the authors propose a lightweight text-denoising adaptation approach that reformulates the audio projection task as a text denoising problem. By training the LLM to reconstruct clean transcripts from noisy textual inputs, the method achieves effective domain adaptation without modifying the model architecture or introducing additional parameters. This strategy preserves cross-modal alignment while significantly improving recognition accuracy, yielding up to a 22.1% relative reduction in word error rate on two benchmark datasets—substantially outperforming current state-of-the-art text-only adaptation techniques.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Adapting automatic speech recognition (ASR) systems based on large language models (LLMs) to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on target-domain text often disrupts the critical alignment between speech and text modalities learned by the projector, degrading performance. We introduce a novel text-only adaptation method that emulates the audio projection task by treating it as a text denoising task. Our approach thus trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.

Problem

• text-only adaptation
• LLM-based ASR
• domain adaptation
• cross-modal alignment
• automatic speech recognition

Innovation

• text-only adaptation
• text denoising
• LLM-based ASR
• cross-modal alignment
• domain adaptation