A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Rater's Shadowing and Sequence-to-sequence Voice Conversion

2025-05-30

AI Summary

This study addresses the limitation of conventional ASR-based approaches for L2 speech intelligibility assessment, which over-rely on native-language similarity and diverge from human perceptual judgments. We propose a perception-driven quantitative paradigm that explicitly models native speakers’ shadowing behavior as an auditory-perceptual alignment signal. Our method jointly encodes acoustic representation and perceptual response by integrating end-to-end voice conversion, dynamic time warping (DTW)-based alignment, and multi-scale mel-spectrogram reconstruction. Compared to state-of-the-art ASR baselines (r = 0.62), our metric achieves significantly higher correlation with human ratings (r = 0.89). Subjective evaluations further confirm its ability to precisely localize L2 comprehension-difficulty segments. This work overcomes nativeness bias and establishes the first interpretable, computationally tractable framework for intelligibility assessment grounded in empirically observed listener perception behavior.

Technology Category

• Natural Language Processing: Speech
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

Evaluating L2 speech intelligibility is crucial for effective computer-assisted language learning (CALL). Conventional ASR-based methods often focus on native-likeness, which may fail to capture the actual intelligibility perceived by human listeners. In contrast, our work introduces a novel, perception based L2 speech intelligibility indicator that leverages a native rater's shadowing data within a sequence-to-sequence (seq2seq) voice conversion framework. By integrating an alignment mechanism and acoustic feature reconstruction, our approach simulates the auditory perception of native listeners, identifying segments in L2 speech that are likely to cause comprehension difficulties. Both objective and subjective evaluations indicate that our method aligns more closely with native judgments than traditional ASR-based metrics, offering a promising new direction for CALL systems in a global, multilingual contexts.

Problem

• Evaluating L2 speech intelligibility for CALL systems
• Overcoming ASR limitations in capturing human perception
• Simulating native listener perception using seq2seq voice conversion

Innovation

• Uses rater's shadowing data for intelligibility
• Seq2seq voice conversion framework
• Alignment and acoustic feature reconstruction