TELEVAL: A Dynamic Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios

2025-07-23

AI Summary

Existing spoken language model (SLM) benchmarks primarily evaluate performance on complex, task-oriented scenarios, neglecting semantic understanding and responsive capabilities essential for natural, user-initiated dialogue. Method: We propose TELEVAL, a dynamic, Chinese-specific evaluation benchmark designed for interactive settings. It assesses SLMs across three dimensions—explicit semantics, paralinguistic and implicit semantics, and system-level capabilities—in instruction-free, multi-turn, multimodal (text + audio) dialogues. TELEVAL employs realistic dialogue formats and dynamic interaction protocols, enabling the first fine-grained evaluation of implicit intent recognition and context-adaptive response generation. Results: Experiments reveal substantial deficiencies in current SLMs’ natural conversational competence. TELEVAL effectively discriminates models based on authentic interactive capability, thereby bridging a critical gap in user-experience-aligned evaluation.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Spoken language models (SLMs) have seen rapid progress in recent years, along with the development of numerous benchmarks for evaluating their performance. However, most existing benchmarks primarily focus on evaluating whether SLMs can perform complex tasks comparable to those tackled by large language models (LLMs), often failing to align with how users naturally interact in real-world conversational scenarios. In this paper, we propose TELEVAL, a dynamic benchmark specifically designed to evaluate SLMs' effectiveness as conversational agents in realistic Chinese interactive settings. TELEVAL defines three evaluation dimensions: Explicit Semantics, Paralinguistic and Implicit Semantics, and System Abilities. It adopts a dialogue format consistent with real-world usage and evaluates text and audio outputs separately. TELEVAL particularly focuses on the model's ability to extract implicit cues from user speech and respond appropriately without additional instructions. Our experiments demonstrate that despite recent progress, existing SLMs still have considerable room for improvement in natural conversational tasks. We hope that TELEVAL can serve as a user-centered evaluation framework that directly reflects the user experience and contributes to the development of more capable dialogue-oriented SLMs.

Problem

• Evaluating SLMs in realistic Chinese conversational scenarios
• Assessing SLMs' ability to extract implicit speech cues
• Aligning SLM benchmarks with natural user interaction patterns

Innovation

• Dynamic benchmark for Chinese SLM evaluation
• Three evaluation dimensions: explicit, implicit, system
• Separate text and audio output assessment