Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

2026-09-15

AI Summary

为解决语音大模型在领域迁移时的适应问题，提出通过在编码层插入轻量级适配器并结合LoRA对LLM进行微调的方法，有效提升了模型在新领域的性能。

Abstract

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.

Problem

• Speech-LLMs
• Domain-Adaptive
• ASR
• Limited-Data
• Acoustic-Adaptation

Innovation

• Encoder Awakening via Adapters
• Domain-Adaptive Fine-Tuning
• Low-Rank Adapters (LoRA)
• Speech-LLM