An Evaluation Framework for Text-to-Speech Voice Reconstruction

2026-06-19

AI Summary

Current evaluation of speech reconstruction relies heavily on Mean Opinion Score (MOS) ratings, which struggle to reliably capture the trade-off between naturalness and speaker similarity for highly unintelligible speech. To address this limitation, this work proposes the first hybrid subjective–objective evaluation framework. Subjectively, it introduces a contextualized Best–Worst Scaling (BWS) paradigm to precisely assess intelligibility and speaker identity perception. Objectively, it designs a novel dual-reference distribution-based metric that effectively quantifies the relationship between these two dimensions. Experiments across 17 zero-shot text-to-speech systems and 193 speakers demonstrate that the proposed framework achieves high reliability and strong alignment with downstream tasks, significantly outperforming existing evaluation methods.

Abstract

Voice reconstruction using Text-to-Speech (TTS) offers a communication method for people with speech disorders, which aims to retain their speaker identity while improving intelligibility. Previous work generally relies on Mean Opinion Score (MOS) to evaluate naturalness and speaker similarity, but this has limited sensitivity and reliability. We propose an evaluation framework with subjective and objective components. Subjectively, we evaluate perceived intelligibility and speaker identity using Best Worst Scaling (BWS) with situational framing. Objectively, we demonstrate that standard measures fail to predict reconstruction success for highly unintelligible speakers, so we introduce a novel dual-reference distributional measure to assess the trade-off between intelligibility and speaker identity. By evaluating the output of 17 zero-shot TTS systems for 193 speakers, we show that our framework provides a reliable and task-aligned approach for assessing voice reconstruction.

Problem

• Text-to-Speech
• Voice Reconstruction
• Speaker Identity
• Intelligibility
• Evaluation Framework

Innovation

• voice reconstruction
• evaluation framework
• dual-reference distributional measure
• zero-shot TTS
• Best Worst Scaling