Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

2026-09-10

AI Summary

研究解决了联邦多语言语音-LLM中因组件间更新范数差异导致的差分隐私预算崩溃问题，提出α-split方法分配独立参数池以改善此情况。

Abstract

Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer \emph{cross-component budget collapse}, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose \emph{$α$-split}, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint $\ell_2$ sensitivity and the original $(\varepsilon,δ)$-DP guarantee are unchanged. At architecture-calibrated $α$, our method recovers WER utility compared to flat DP, while granting the encoder $4.47{\times}$ tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only $+2.6\%$ LLM noise overhead.

Problem

• federated learning
• differential privacy
• speech-LLMs
• cross-component budget collapse
• word error rate

Innovation

• α-split
• differential privacy
• federated learning
• speech-LLMs
• cross-component budget collapse