What do Speech Foundation Models Learn? Analysis and Applications

2025-08-17

AI Summary

The knowledge composition of speech foundation models (SFMs) and their suitability for spoken language understanding (SLU) remain poorly understood, and there is a lack of fine-grained, task-specific evaluation benchmarks. Method: We propose a lightweight, training-free analytical framework that employs statistical probes and parameter-free tasks to systematically dissect the distribution of acoustic and linguistic knowledge across SFM layers. Based on this analysis, we introduce SpokenNER—the first benchmark for spoken named entity recognition and localization—enabling fine-grained knowledge localization. Contribution/Results: Our evaluation spans both self-supervised and supervised SFMs, revealing consistent hierarchical knowledge encoding patterns across architectures. Experiments demonstrate that end-to-end SFMs significantly outperform traditional cascade approaches on SLU tasks, validating their strong potential for direct SLU deployment. The framework provides an interpretable tool for probing model internals, while SpokenNER establishes a practical, fine-grained evaluation standard for spoken language understanding.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Speech foundation models (SFMs) are designed to serve as general-purpose representations for a wide range of speech-processing tasks. The last five years have seen an influx of increasingly successful self-supervised and supervised pre-trained models with impressive performance on various downstream tasks. Although the zoo of SFMs continues to grow, our understanding of the knowledge they acquire lags behind. This thesis presents a lightweight analysis framework using statistical tools and training-free tasks to investigate the acoustic and linguistic knowledge encoded in SFM layers. We conduct a comparative study across multiple SFMs and statistical tools. Our study also shows that the analytical insights have concrete implications for downstream task performance. The effectiveness of an SFM is ultimately determined by its performance on speech applications. Yet it remains unclear whether the benefits extend to spoken language understanding (SLU) tasks that require a deeper understanding than widely studied ones, such as speech recognition. The limited exploration of SLU is primarily due to a lack of relevant datasets. To alleviate that, this thesis contributes tasks, specifically spoken named entity recognition (NER) and named entity localization (NEL), to the Spoken Language Understanding Evaluation benchmark. We develop SFM-based approaches for NER and NEL, and find that end-to-end (E2E) models leveraging SFMs can surpass traditional cascaded (speech recognition followed by a text model) approaches. Further, we evaluate E2E SLU models across SFMs and adaptation strategies to assess the impact on task performance. Collectively, this thesis tackles previously unanswered questions about SFMs, providing tools and datasets to further our understanding and to enable the community to make informed design choices for future model development and adoption.

Problem

• Analyze acoustic and linguistic knowledge in speech foundation models
• Explore SFM effectiveness in spoken language understanding tasks
• Develop datasets and tools for SFM evaluation and improvement

Innovation

• Lightweight analysis framework for SFM knowledge
• SFM-based end-to-end spoken language understanding
• Contribution of NER and NEL tasks to SLU benchmark