Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion

2026-02-28

AI Summary

This work addresses the excessive key-value (KV) cache memory consumption of the Whisper model in long-form automatic speech recognition, which stems from its multi-head attention (MHA) mechanism and hinders efficient deployment. To mitigate this, the authors propose the first integration of Multi-head Latent Attention (MLA) into Whisper by replacing the original MHA in the decoder self-attention modules and adapting it to Whisper’s absolute positional encoding scheme. Systematic experiments demonstrate that applying MLA solely within the decoder self-attention reduces KV cache usage by up to 87.5% on LibriSpeech while maintaining competitive word error rates. This architectural modification substantially enhances inference efficiency and memory economy without compromising recognition accuracy.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy.

Problem

• Automatic Speech Recognition
• GPU memory consumption
• Multi-Head Attention
• KV cache
• long-form audio

Innovation

• Multi-Head Latent Attention
• KV cache compression
• Whisper model
• Automatic Speech Recognition
• Memory-efficient ASR