Beyond Words: Measuring User Experience through Speech Analysis in Voice User Interfaces

2026-03-20

AI Summary

This study addresses the limitations of conventional user experience (UX) evaluation methods for voice assistants, which predominantly rely on task completion rates and subjective questionnaires and thus fail to capture users’ genuine emotions and interaction difficulties in real time. To overcome this, the work proposes a novel implicit evaluation paradigm that requires no active user feedback by systematically investigating the relationship between paralinguistic vocal features and UX. The authors extract time-domain, spectral, and linguistic speech features and employ machine learning models to construct a voice-based UX classifier. Experimental results demonstrate that multiple vocal features are significantly correlated with user satisfaction, and the proposed model effectively differentiates among distinct UX levels with strong classification accuracy.

Technology Category

• Humans and AI: User Experience and Usability
• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Voice assistants (VAs) are typically evaluated through task performance metrics and self-report questionnaires, but people's voices themselves carry rich paralinguistic cues that reveal affect, effort, and interaction breakdowns. We present a within-subjects study (N=49) that systematically compared three VA personas across three usage scenarios to investigate whether speech-derived audio features can serve as a proxy for user experience (UX). Participants' speech was analyzed for temporal, spectral, and linguistic markers, alongside standardized UX measures, brief mood and stress ratings, and a post-study questionnaire. We found correlations between specific speech features and self-reported satisfaction and experience. Furthermore, a machine learning model trained on speech features achieved promising accuracy in classifying UX levels, indicating that this might be a reasonable alternative to self-report instruments. Our findings establish speech as a viable, real-time signal for implicitly measuring UX and point toward adaptive VUIs that respond dynamically to emotional and usability-related vocal cues.

Problem

• user experience
• speech analysis
• voice user interfaces
• paralinguistic cues
• implicit measurement

Innovation

• speech analysis
• user experience
• voice user interfaces
• paralinguistic cues
• implicit UX measurement