Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR

2026-07-13

AI Summary

This work addresses the challenge of catastrophic forgetting and optimization bias in multilingual low-resource automatic speech recognition (ASR) fine-tuning, where dominant languages often overshadow others. The authors propose Unified Gradient Projection (UGP), a novel approach that introduces language-balanced mechanisms into the gradient projection space for the first time. By generating reference gradients through language-balanced replay and constraining parameter updates within a unified projection space, UGP equitably balances each language’s contribution to optimization. The method synergistically integrates gradient-level projection with data-level replay, enhancing both model stability and plasticity. Evaluated across diverse low-resource language combinations and various scales of pretrained ASR models—including Whisper-large-v3—UGP substantially mitigates forgetting, achieving near-zero average forgetting rates.

Abstract

Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.

Problem

• catastrophic forgetting
• multilingual ASR
• low-resource languages
• cross-task interference
• dominant-language bias

Innovation

• Unified Gradient Projection
• Continual Learning
• Multilingual ASR
• Catastrophic Forgetting
• Language-Balanced Replay