Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages

2026-04-23

AI Summary

This work addresses the high variance and multidimensional perceptual complexity in evaluating multilingual text-to-speech (TTS) systems for Indian languages by proposing the first scalable, language-controllable evaluation framework. Through large-scale crowdsourced pairwise preference experiments across ten Indian languages—encompassing over 5,000 utterances and 120,000 judgments—the study integrates multidimensional perceptual annotations, Bradley–Terry modeling, and SHAP-based interpretability analysis. This approach reveals, for the first time, the performance trade-offs of TTS models across six key dimensions: intelligibility, expressiveness, audio quality, naturalness, speaker similarity, and linguistic accuracy. The project establishes the first human preference leaderboard for Indian multilingual TTS, validates evaluation reliability, and provides an interpretable, fine-grained benchmark for future research in multilingual speech synthesis.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.

Problem

• Text-to-Speech
• pairwise evaluation
• multilingual TTS
• speech perception
• Indian languages

Innovation

• multidimensional pairwise evaluation
• multilingual TTS
• perceptual annotation
• Bradley-Terry modeling
• SHAP analysis