Speejis: Enhancing User Experience of Mobile Voice Messaging with Automatic Visual Speech Emotion Cues

2025-02-07

AI Summary

Current mobile voice messaging lacks automated visual affective expression mechanisms, limiting both the convenience and richness of emotional communication. To address this, we propose *speejis*: a fine-grained, real-time visual affective prompting method for voice messages. Speejis extracts paralinguistic features and models affect continuously along the valence–arousal dimensions to accurately identify and symbolically map dynamic emotional shifts within a single utterance. It is the first approach to convert continuous affective trajectories into lightweight, interpretable visual symbols—enabling accessible affective expression and perception. A user study (N=12) demonstrates that speejis significantly improves key UX metrics—including appeal and stimulation—with unanimous participant preference. This work establishes a novel paradigm and a practical technical pathway for affective visualization in voice-based human–computer interaction.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Speech
• Humans and AI: Interaction Techniques and Devices

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Web Mining and Content Analysis: Web data visualization

Abstract

Mobile messaging apps offer an increasing range of emotional expressions, such as emojis to help users manually augment their texting experiences. Accessibility of such augmentations is limited in voice messaging. With the term"speejis"we refer to accessible emojis and other visual speech emotion cues that are created automatically from speech input alone. The paper presents an implementation of speejis and reports on a user study (N=12) comparing the UX of voice messaging with and without speejis. Results show significant differences in measures such as attractiveness and stimulation and a clear preference of all participants for messaging with speejis. We highlight the benefits of using paralinguistic speech processing and continuous emotion models to enable finer grained augmentations of emotion changes and transitions within a single message in addition to augmentations of the overall tone of the message.

Problem

• Enhancing voice messaging with visual emotion cues
• Automating emotion expression from speech input
• Improving user experience in mobile messaging apps

Innovation

• Automatic visual speech emotion cues
• Paralinguistic speech processing integration
• Continuous emotion models for augmentation