Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment

2026-07-09

AI Summary

This work addresses the alignment bias in Best-of-N text-to-speech synthesis, where the performance of automatic speech recognition (ASR)-based verifiers is confounded by their coupling with the ASR model family used for evaluation. The study is the first to identify this family-dependent coupling effect and proposes two cross-family ranking ensemble strategies—rank-averaging and conjunctive max-rank—that integrate diverse ASR families, including Whisper, wav2vec 2.0, and HuBERT, for zero-shot candidate selection. Experimental results demonstrate that the proposed approach reduces the average word error rate (WER) to 1.61% at N=10, a 12% improvement over the F5-TTS baseline. Furthermore, the best single verifier achieves a WER reduction from 2.06% to 1.72% (↓16.5%) under the official evaluator while maintaining stable SIM-o and UTMOS speech quality metrics.

Abstract

Best-of-$N$ (BoN) inference improves content consistency in zero-shot text-to-speech by selecting from $N$ candidates with an automatic speech recognition (ASR) verifier. We identify an underexplored evaluation confound: a verifier's apparent quality depends strongly on which ASR family judges it. On LibriSpeech-PC test-clean~\citep{librispeechpc} with F5-TTS~\citep{f5tts}, verifier rankings reverse across Whisper, wav2vec~2.0, and HuBERT evaluators, and same-family verifier-evaluator pairs recover 2-3$\times$ more oracle headroom than cross-family pairs despite near-identical representations (linear CKA $0.978$) -- a pattern consistent with identity- or lineage-level coupling rather than representational overlap. We propose two \textbf{cross-family rank ensembles} (rank-averaging and conjunctive max-rank) that attain the lowest mean WER across three independent evaluators -- $1.61\%$ at $N{=}10$ ($-12\%$ relative to F5-TTS) -- with no measurable degradation under automatic SIM-o/UTMOS metrics; the best single verifier drives WER from $2.06\%$ to $1.72\%$ ($-16.5\%$) under the official F5-TTS evaluator. We recommend cross-evaluator triangulation as default reporting practice.

Problem

• Best-of-N TTS
• ASR family alignment
• evaluation confound
• zero-shot text-to-speech
• verifier-evaluator coupling

Innovation

• Best-of-N TTS
• ASR family alignment
• cross-family rank ensemble
• evaluation confounding
• zero-shot text-to-speech