PRiSM: Benchmarking Phone Realization in Speech Models

2026-01-20

AI Summary

Current evaluations of phoneme recognition systems are largely confined to surface-level transcription accuracy, with limited insight into their underlying phonemic perception capabilities. This work proposes PRiSM—the first open-source comprehensive benchmark for phoneme recognition—which establishes a standardized evaluation framework integrating both intrinsic (representation probing) and extrinsic (downstream tasks across clinical, educational, and multilingual settings) assessments. Leveraging transcription-based metrics, multilingual datasets, and encoder-CTC architectures, the benchmark enables reproducible evaluation and reveals that multilingual training substantially enhances performance, encoder-CTC models exhibit the most consistent results, and specialized phoneme recognition models still outperform general-purpose large audio language models.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.

Problem

• Phone Recognition
• Speech Models
• Phonetic Perception
• Multilingual Speech Processing
• Benchmarking

Innovation

• Phone Recognition
• Benchmarking
• Multilingual Speech Modeling
• Phonetic Evaluation
• Audio Language Models