🤖 AI Summary
To address the limited generalization capability of self-supervised pretraining for low-resource languages (e.g., Russian) in automatic speech recognition (ASR), this paper proposes GigaAM: (1) a recognition-oriented joint self-supervised objective that unifies masked language modeling with knowledge distillation signals from ASR models; (2) a novel dynamic chunked attention mechanism, enabling variable chunk-size sampling to simultaneously support full-context offline inference and low-latency streaming fine-tuning; and (3) large-scale model scaling during training, yielding the open-source GigaAM model family released under the MIT License. On Russian ASR benchmarks, GigaAM reduces word error rate (WER) by 50% relative to Whisper-large-v3, demonstrating substantial improvements in robustness for low-resource languages and flexibility for diverse deployment scenarios.
📝 Abstract
Self-Supervised Learning (SSL) has demonstrated strong performance in speech processing, particularly in automatic speech recognition. In this paper, we explore an SSL pretraining framework that leverages masked language modeling with targets derived from a speech recognition model. We also present chunkwise attention with dynamic chunk size sampling during pretraining to enable both full-context and streaming fine-tuning. Our experiments examine scaling with respect to model size and the amount of data. Using our method, we train the GigaAM family of models, including a state-of-the-art model for Russian speech recognition that outperforms Whisper-large-v3 by 50%. We have released our foundation and ASR models, along with the inference code, under the MIT license as open-source resources to the research community. Available at https://github.com/salute-developers/gigaam.