Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

2026-04-11

AI Summary

This work addresses the limitation of current general-purpose audio large language models, which rely on Whisper encoders trained exclusively on speech and thus struggle to effectively represent music and environmental sounds, leading to high training costs for downstream tasks. The study presents the first extension of Whisper-large-v3 to non-speech domains by performing end-to-end fine-tuning on a mixed dataset encompassing speech, environmental sounds, and music, yielding a multi-domain compatible enhanced encoder that can be seamlessly integrated as a plug-and-play module into audio large language models. Experimental results demonstrate substantial cross-domain performance gains under linear probing: accuracy improves by 23.0% on ESC-50 environmental sound classification, 5.0% on GTZAN music genre recognition, and 0.7% on Speech Commands keyword spotting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Audio-native large language models (audio-LLMs) commonly use Whisper as their audio encoder. However, Whisper was trained exclusively on speech data, producing weak representations for music and environmental sound. This forces downstream audio-LLMs to compensate through extensive training on large-scale non-speech data. We present Whisper-AuT, a domain-adapted audio encoder obtained by fine-tuning Whisper-large-v3 on a curated mixture of speech (80%), environmental sound (10%), and music (10%) totaling approximately 20M samples. The full encoder-decoder is trained end-to-end with a seq2seq captioning objective; the decoder is then discarded and only the encoder is retained. Linear probe evaluations show that Whisper-AuT achieves +23.0% on ESC-50 (environmental sound), +5.0% on GTZAN (music genre), and +0.7% on Speech Commands (keyword spotting) compared to the original Whisperlarge-v3 encoder. Whisper-AuT is designed as a drop-in replacement for Whisper in audio-LLM architectures, with the goal of reducing downstream training cost by providing stronger initial audio representations for non-speech domains.

Problem

• audio-LLM
• Whisper
• domain adaptation
• non-speech audio
• audio representation

Innovation

• domain adaptation
• audio encoder
• Whisper-AuT
• audio-LLM
• multi-domain audio representation