Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR

2026-07-06

AI Summary

This work addresses the limited capacity of end-to-end automatic speech recognition (ASR) models to iteratively refine predictions for challenging inputs. The authors propose LatentASR, a method that enables efficient and stable test-time adaptation through continuous latent variables while keeping the backbone model parameters frozen. Requiring only approximately 4M additional parameters and a small activation set of 500 utterances, LatentASR employs a latent adapter for dynamic updates and integrates a value head to implement an input-dependent early stopping strategy. The approach achieves consistent performance gains across diverse benchmarks: relative word error rate (WER) reductions of 2.54% and 0.47% on FLEURS and VoxPopuli, respectively, and a 16.0% relative character error rate (CER) improvement on ASCEND data featuring accented speech and code-switching, with consistent enhancements observed across 30 languages.

Abstract

End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.

Problem

• automatic speech recognition
• test-time scaling
• hard inputs
• code-switched speech
• accented speech

Innovation

• test-time scaling
• latent adapter
• parameter-efficient adaptation
• dynamic halting
• frozen ASR backbone