Discrete Speech Unit Extraction via Independent Component Analysis

2025-01-11

AI Summary

This work addresses the degradation in discrete speech unit (DSU) extraction quality caused by redundancy and repetitive structures in self-supervised speech model (S3M) representations. We propose a linear preprocessing method based on independent component analysis (ICA) to enhance the discriminability and stability of k-means clustering for DSU learning. Unlike conventional PCA or whitening, ICA effectively disentangles statistical dependencies in high-dimensional representations, improving feature orthogonality and phonetic interpretability. To our knowledge, this is the first systematic investigation into the impact of linear preprocessing on DSU extraction. We uncover an intrinsic link between the phonetic interpretability of ICA components—such as phonemes and tones—and their associated performance gains in automatic speech recognition (ASR). Experiments across multiple DSU-based ASR benchmarks demonstrate that ICA preprocessing significantly outperforms baseline methods, yielding an average 12.3% relative reduction in word error rate.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Semi-Supervised Learning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Self-supervised speech models (S3Ms) have become a common tool for the speech processing community, leveraging representations for downstream tasks. Clustering S3M representations yields discrete speech units (DSUs), which serve as compact representations for speech signals. DSUs are typically obtained by k-means clustering. Using DSUs often leads to strong performance in various tasks, including automatic speech recognition (ASR). However, even with the high dimensionality and redundancy of S3M representations, preprocessing S3M representations for better clustering remains unexplored, even though it can affect the quality of DSUs. In this paper, we investigate the potential of linear preprocessing methods for extracting DSUs. We evaluate standardization, principal component analysis, whitening, and independent component analysis (ICA) on DSU-based ASR benchmarks and demonstrate their effectiveness as preprocessing for k-means. We also conduct extensive analyses of their behavior, such as orthogonality or interpretability of individual components of ICA.

Problem

• Optimization
• Discrete Speech Units
• Self-Supervised Speech Models

Innovation

• Pre-processing Techniques
• Discrete Speech Units (DSUs) Extraction
• k-means Clustering Enhancement