HPSU: A Benchmark for Human-Level Perception in Real-World Spoken Speech Understanding

2025-11-28

AI Summary

Current speech large language models (SLLMs) perform well on explicit speech understanding tasks (e.g., ASR, SER) but lack systematic evaluation of human-level auditory perception—particularly implicit intent recognition and latent emotion detection in natural spoken discourse. To address this gap, we introduce HPSU, the first benchmark for human-level perceptual speech understanding, featuring a multi-tiered evaluation framework spanning speaker attribute identification to complex semantic inference. HPSU employs a cross-modal (audio–text–vision) semi-automated annotation paradigm, augmented by expert validation, to significantly improve both annotation efficiency and fidelity. Comprehensive evaluation on HPSU reveals substantial performance gaps between state-of-the-art SLLMs and human annotators in implicit semantic understanding, exposing critical deficiencies in modeling contextual dependency, non-literal expressions, and interactive dynamics.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Recent advances in Speech Large Language Models (Speech LLMs) have led to great progress in speech understanding tasks such as Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). However, whether these models can achieve human-level auditory perception, particularly in terms of their ability to comprehend latent intentions and implicit emotions in real-world spoken language, remains underexplored. To this end, we introduce the Human-level Perception in Spoken Speech Understanding (HPSU), a new benchmark for fully evaluating the human-level perceptual and understanding capabilities of Speech LLMs. HPSU comprises over 20,000 expert-validated spoken language understanding samples in English and Chinese. It establishes a comprehensive evaluation framework by encompassing a spectrum of tasks, ranging from basic speaker attribute recognition to complex inference of latent intentions and implicit emotions. To address the issues of data scarcity and high cost of manual annotation in real-world scenarios, we developed a semi-automatic annotation process. This process fuses audio, textual, and visual information to enable precise speech understanding and labeling, thus enhancing both annotation efficiency and quality. We systematically evaluate various open-source and proprietary Speech LLMs. The results demonstrate that even top-performing models still fall considerably short of human capabilities in understanding genuine spoken interactions. Consequently, HPSU will be useful for guiding the development of Speech LLMs toward human-level perception and cognition.

Problem

• Evaluating Speech LLMs' human-level perception in real-world spoken language understanding
• Assessing comprehension of latent intentions and implicit emotions in speech
• Addressing data scarcity and annotation challenges for speech understanding tasks

Innovation

• Semi-automatic annotation process fuses multimodal information
• Comprehensive evaluation framework for human-level auditory perception
• Benchmark with 20000 expert-validated multilingual speech samples