SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models

2025-07-25

AI Summary

Current speech evaluation overly relies on Word Error Rate (WER), failing to reflect models’ true speech understanding capabilities in memory, comprehension, and application. To address this, we propose Speech-based Intelligence Quotient (SIQ)—the first three-tier speech understanding evaluation framework inspired by Bloom’s Taxonomy, uniformly applicable to both cascaded and end-to-end models. SIQ integrates automatic speech recognition accuracy, semantic similarity, and downstream question-answering performance to enable quantifiable, cognition-grounded assessment. Experiments demonstrate that SIQ effectively uncovers annotation errors and model hallucinations in mainstream benchmarks, reveals latent deficiencies in multimodal training, and enables fair cross-architecture model comparison. By grounding evaluation in cognitive principles, SIQ significantly enhances the conceptual validity and practical utility of speech intelligence assessment.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training.

Problem

• Assessing voice understanding in LLMs using cognitive levels
• Moving beyond WER to evaluate LLM Voice performance
• Unifying comparisons between cascaded and end-to-end models

Innovation

• Introduces Speech-based Intelligence Quotient (SIQ) evaluation
• Assesses LLM Voice across three cognitive levels
• Unifies comparisons between cascaded and end-to-end models