Investigation for Relative Voice Impression Estimation

2026-02-15

AI Summary

This study addresses the limitation of traditional voice impression assessment, which relies on absolute ratings and struggles to capture subtle perceptual differences between speech segments from the same speaker. To overcome this, the authors propose a Relative Voice Impression Estimation (RIE) framework that quantifies expressive and prosodic variations by modeling perceived shifts along antonymic dimensions (e.g., “dark–bright”) within paired utterances from the same source. The work presents the first systematic investigation of this task, introducing a novel paired dataset based on multi-style readings by professional broadcasters. Comprehensive comparisons are conducted among classical acoustic features, self-supervised speech representations, and multimodal large language models. Results demonstrate that approaches leveraging self-supervised representations significantly outperform traditional methods, particularly on complex impression dimensions such as “cold–warm,” while multimodal large language models show limited efficacy in this fine-grained perceptual task.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Paralinguistic and non-linguistic aspects of speech strongly influence listener impressions. While most research focuses on absolute impression scoring, this study investigates relative voice impression estimation (RIE), a framework for predicting the perceptual difference between two utterances from the same speaker. The estimation target is a low-dimensional vector derived from subjective evaluations, quantifying the perceptual shift of the second utterance relative to the first along an antonymic axis (e.g., ``Dark--Bright''). To isolate expressive and prosodic variation, we used recordings of a professional speaker reading a text in various styles. We compare three modeling approaches: classical acoustic features commonly used for speech emotion recognition, self-supervised speech representations, and multimodal large language models (MLLMs). Our results demonstrate that models using self-supervised representations outperform methods with classical acoustic features, particularly in capturing complex and dynamic impressions (e.g., ``Cold--Warm'') where classical features fail. In contrast, current MLLMs prove unreliable for this fine-grained pairwise task. This study provides the first systematic investigation of RIE and demonstrates the strength of self-supervised speech models in capturing subtle perceptual variations.

Problem

• relative voice impression estimation
• paralinguistic perception
• perceptual shift
• antonymic axis
• expressive variation

Innovation

• relative voice impression estimation
• self-supervised speech representations
• paralinguistic perception
• antonymic axis
• expressive prosody