WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

2026-03-17

arXiv.org

AI Summary

This work addresses the high computational and memory costs of autoregressive text-to-speech (AR-TTS) models, which stem from full self-attention mechanisms whose complexity grows quadratically with sequence length, hindering efficient deployment. To overcome this limitation, the authors propose the WAND framework—the first to integrate sliding window attention for local token modeling, global conditioning attention, curriculum learning-based fine-tuning, and efficient knowledge distillation within AR-TTS, alongside optimized key-value (KV) caching. This approach reduces inference complexity to be independent of sequence length, achieving near-constant per-step latency while preserving speech quality without degradation across three prominent AR-TTS architectures. The method substantially cuts KV cache memory usage by up to 66.2%, enabling significantly more efficient real-time synthesis.

Abstract

Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.

Problem

• autoregressive text-to-speech
• self-attention
• computational complexity
• memory efficiency
• sequence length

Innovation

• Windowed Attention
• Knowledge Distillation
• Autoregressive TTS
• Constant Complexity
• Curriculum Learning