Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models

2026-06-12

AI Summary

Current explanation methods for Transformer-based speech recognition models suffer from insufficient faithfulness and limited temporal localization accuracy. This work proposes LEAF-X, a novel framework that, for the first time, integrates an entropy-guided mechanism into attention analysis. By combining multi-layer attention rollout, causal ablation, and encoder-decoder architecture, LEAF-X generates token-to-frame attributions that are sparse, highly faithful, and strongly localized in time. Experimental results demonstrate that LEAF-X improves faithfulness by 32% over baseline methods, enhances locality and sparsity by 35–39%, and substantially increases attribution stability.

Abstract

Transformer-based automatic speech recognition (ASR) models such as Whisper are highly accurate, but their predictions remain difficult to interpret. Existing explainable AI (XAI) methods often lack faithfulness and precise temporal grounding. We propose Listening with Entropy-guided Attention for Faithful explainability (LEAF-X), a model-intrinsic XAI framework for transformer-based ASR. LEAF-X combines entropy-guided attention weighting, multi-layer attention rollout, and optional causal ablations to identify low-entropy, high-impact heads and layers, producing sparse token-to-frame attributions. Unlike perturbation-based explainers or raw attention maps, LEAF-X exploits the internal structure of encoder-decoder and speech-augmented decoder-only models to generate explanations that better reflect model computation. Results show 32% improved faithfulness, 35-39% stronger locality/sparsity, and the most stable attributions, supporting more transparent and auditable ASR.

Problem

• explainable AI
• automatic speech recognition
• Transformer models
• faithfulness
• temporal grounding

Innovation

• entropy-guided attention
• faithful explainability
• transformer-based ASR
• attention rollout
• token-to-frame attribution