Is Natural Always Appropriate? Investigating Naturalness and Appropriateness Across Different Domains for TTS Evaluation

2026-06-30

AI Summary

Current text-to-speech (TTS) evaluation practices place excessive emphasis on naturalness while largely neglecting the appropriateness of synthesized speech across diverse application contexts. This study conducts human subjective evaluations of five state-of-the-art TTS systems across five distinct scenarios—AI assistants, narrators, actors, animated characters, and spontaneous speakers—systematically revealing, for the first time, a decoupling between naturalness and contextual appropriateness in multi-domain TTS. The findings indicate that while TTS excels in read-aloud tasks, it remains challenged in highly expressive settings. Moreover, optimizing for a single dimension can inadvertently degrade performance in others, and naturalness scores often misjudge stylized speech. These results highlight significant blind spots in conventional evaluation metrics when applied to expressive or context-sensitive speech synthesis.

Abstract

Text-to-speech (TTS) evaluation is an open challenge. While the primary target was "naturalness," recent fidelity gains shifted focus toward "appropriateness" and whether speech is correct for its context. In this work, we examine how perception changes when the expected downstream use varies. We measure the appropriateness and human-likeness of five SOTA TTS systems across five domains: AI assistant, reader, actor, animated character, and spontaneous speaker. Results show appropriateness varies across domains independently of naturalness. While systems shine at reading, expressive domains remain challenging, and optimizing for one can degrade others. Furthermore, naturalness scores tend to penalize stylized speech while rewarding spontaneity. Finally, our study also highlights blind spots in one-size-fits-all evaluation metrics across more expressive domains. We demonstrate that TTS performance is not "solved" but depends on the target domain, requiring context-aware evaluation.

Problem

• TTS evaluation
• naturalness
• appropriateness
• domain-specific
• expressive speech

Innovation

• appropriateness
• naturalness
• domain-specific evaluation
• text-to-speech
• context-aware assessment