SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation

2026-04-22

AI Summary

This study addresses the lack of fine-grained, objective evaluation frameworks for paralinguistic feature generation in current large audio language models (LALMs). The authors introduce the first comprehensive benchmark encompassing over 100 fine-grained paralinguistic attributes, built upon more than 1,000 English–Chinese parallel spoken queries, and structured around three progressively complex tasks: fine-grained control, intra-utterance variation, and context-adaptive modulation. They further propose an innovative pairwise comparison evaluation protocol leveraging LALMs themselves, which replaces absolute scoring with relative preference judgments to substantially reduce subjectivity and annotation costs. Experimental results reveal significant deficiencies in mainstream models’ ability to statically control and dynamically modulate paralinguistic features, with 43.3% of contextual dialogue errors attributable to failures in interpreting paralinguistic cues—highlighting the urgent need for focused research in this area.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Paralinguistic cues are essential for natural human-computer interaction, yet their evaluation in Large Audio-Language Models (LALMs) remains limited by coarse feature coverage and the inherent subjectivity of assessment. To address these challenges, we introduce SpeechParaling-Bench, a comprehensive benchmark for paralinguistic-aware speech generation. It expands existing coverage from fewer than 50 to over 100 fine-grained features, supported by more than 1,000 English-Chinese parallel speech queries, and is organized into three progressively challenging tasks: fine-grained control, intra-utterance variation, and context-aware adaptation. To enable reliable evaluation, we further develop a pairwise comparison pipeline, in which candidate responses are evaluated against a fixed baseline by an LALM-based judge. By framing evaluation as relative preference rather than absolute scoring, this approach mitigates subjectivity and yields more stable and scalable assessments without costly human annotation. Extensive experiments reveal substantial limitations in current LALMs. Even leading proprietary models struggle with comprehensive static control and dynamic modulation of paralinguistic features, while failure to correctly interpret paralinguistic cues accounts for 43.3% of errors in situational dialogue. These findings underscore the need for more robust paralinguistic modeling toward human-aligned voice assistants.

Problem

• paralinguistic cues
• speech generation
• Large Audio-Language Models
• evaluation benchmark
• subjectivity

Innovation

• paralinguistic-aware speech generation
• SpeechParaling-Bench
• fine-grained control
• pairwise comparison pipeline
• Large Audio-Language Models