From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales

2026-03-31

arXiv.org

AI Summary

This work addresses the critical safety concerns posed by hallucination in large-scale automatic speech recognition (ASR) systems, whose underlying mechanisms remain poorly understood. The study introduces the Spectral Sensitivity Theorem, establishing—for the first time—a theoretical link between ASR hallucination and the spectral dynamics of attention mechanisms, revealing a phase transition from dispersive to attractive states driven by inter-layer gain and alignment. Through spectral analysis, eigen-decomposition of activation maps, adversarial perturbations, and dynamic tracking of attention ranks, the authors identify two distinct failure modes: medium-scale models exhibit a 13.4% cross-attention rank collapse (“structural disintegration”), while large-scale models enter a “compressed attractor” state, characterized by a 2.34% reduction in self-attention rank, spectral slope hardening, and decoupling from acoustic evidence. These findings provide a foundational theoretical framework for understanding and mitigating ASR hallucinations.

Abstract

Hallucinations in large ASR models present a critical safety risk. In this work, we propose the \textit{Spectral Sensitivity Theorem}, which predicts a phase transition in deep networks from a dispersive regime (signal decay) to an attractor regime (rank-1 collapse) governed by layer-wise gain and alignment. We validate this theory by analyzing the eigenspectra of activation graphs in Whisper models (Tiny to Large-v3-Turbo) under adversarial stress. Our results confirm the theoretical prediction: intermediate models exhibit \textit{Structural Disintegration} (Regime I), characterized by a $13.4\%$ collapse in Cross-Attention rank. Conversely, large models enter a \textit{Compression-Seeking Attractor} state (Regime II), where Self-Attention actively compresses rank ($-2.34\%$) and hardens the spectral slope, decoupling the model from acoustic evidence.

Problem

• hallucination
• automatic speech recognition
• Whisper models
• spectral dynamics
• model scale

Innovation

• Spectral Sensitivity Theorem
• hallucination dynamics
• rank collapse
• attention mechanism
• phase transition