StressTest: Can YOUR Speech LM Handle the Stress?

2025-05-28

AI Summary

Current speech-language models (SLMs) lack explicit modeling of sentence stress, hindering their ability to discern semantic and pragmatic differences induced by stress variation. To address this, we introduce StressTest—the first benchmark dedicated to sentence-stress understanding—and develop a phoneme-aligned, stress-controllable synthetic data generation pipeline, releasing the Stress17k training dataset. We further propose StresSLM, an end-to-end differentiable SLM architecture integrating phoneme-level alignment, stress-conditioned speech synthesis, adversarial audio augmentation, and multi-task joint optimization. Experiments demonstrate that StresSLM achieves substantial gains over state-of-the-art methods: +31.4% absolute improvement in stress inference accuracy and +23.6% in stress detection accuracy. Moreover, it exhibits strong robustness across both real-world recordings and synthetic speech, validating its generalizability and practical utility for stress-aware language understanding.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Sentence stress refers to emphasis, placed on specific words within a spoken utterance to highlight or contrast an idea, or to introduce new information. It is often used to imply an underlying intention that is not explicitly stated. Recent advances in speech-aware language models (SLMs) have enabled direct processing of audio, allowing models to bypass transcription and access the full richness of the speech signal and perform audio reasoning tasks such as spoken question answering. Despite the crucial role of sentence stress in shaping meaning and speaker intent, it remains largely overlooked in evaluation and development of such models. In this work, we address this gap by introducing StressTest, a benchmark specifically designed to evaluate a model's ability to distinguish between interpretations of spoken sentences based on the stress pattern. We assess the performance of several leading SLMs and find that, despite their overall capabilities, they perform poorly on such tasks. To overcome this limitation, we propose a novel synthetic data generation pipeline, and create Stress17k, a training set that simulates change of meaning implied by stress variation. Then, we empirically show that optimizing models with this synthetic dataset aligns well with real-world recordings and enables effective finetuning of SLMs. Results suggest, that our finetuned model, StresSLM, significantly outperforms existing models on both sentence stress reasoning and detection tasks. Code, models, data, and audio samples - pages.cs.huji.ac.il/adiyoss-lab/stresstest.

Problem

• Evaluating speech LMs' ability to interpret sentence stress patterns
• Addressing poor performance of SLMs in stress-based meaning distinction
• Proposing synthetic data to improve stress reasoning in SLMs

Innovation

• Introduces StressTest benchmark for stress evaluation
• Proposes synthetic data generation pipeline Stress17k
• Develops finetuned model StresSLM for stress tasks