Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

2024-01-18

IEEE International Conference on Acoustics, Speech, and Signal Processing

AI Summary

To address the high communication overhead and performance degradation caused by client data heterogeneity in federated learning (FL) for automatic speech recognition (ASR) and speech translation (ST), this paper proposes FedLoRA-Mem, a lightweight personalized FL framework. Methodologically, it integrates two core components: (1) FedLoRA, which introduces Low-Rank Adaptation (LoRA) adapters into client-side fine-tuning—markedly reducing uplink parameter transmission—and (2) FedMem, a global model augmented with a k-nearest neighbors (kNN) memory mechanism to enhance personalized representation learning. Built upon Conformer or Whisper backbones, FedLoRA-Mem synergistically combines federated averaging with local adapter-based adaptation. Evaluated on CoVoST and GigaSpeech benchmarks, it achieves substantial reductions in communication cost while consistently outperforming baselines such as FedAvg in both WER (ASR) and BLEU (ST), effectively mitigating the adverse effects of statistical heterogeneity.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems, including automatic speech recognition (ASR) and speech translation (ST). However, the commonly used FL approach (i.e., FEDAVG) in S2T tasks typically suffers from extensive communication overhead due to multi-round interactions based on the whole model and performance degradation caused by data heterogeneity among clients. To address these issues, we propose a personalized federated S2T framework that introduces FEDLORA, a lightweight LoRA module for client-side tuning and interaction with the server to minimize communication overhead, and FEDMEM, a global model equipped with a k-nearest-neighbor (kNN) classifier that captures client-specific distributional shifts to achieve personalization and overcome data heterogeneity. Extensive experiments based on Conformer and Whisper backbone models on CoVoST and GigaSpeech benchmarks show that our approach significantly reduces the communication overhead on all S2T tasks and effectively personalizes the global model to overcome data heterogeneity.

Problem

• Federated Learning
• Speech-to-Text Systems
• Communication Overhead

Innovation

• FedLoRA
• FedMem
• Personalized Learning Framework