Problem
Research questions and friction points this paper is trying to address.
Federated Learning
Regularization
Flat Minima
Non-IID Data
Update Masking
How Much Regularization Survives Averaging? Update Masking in Federated Learning
🤖 AI Summary
研究探讨了联邦学习中通过更新掩码方法来实现平坦极小值的问题，分析了不同掩码策略对优化器的影响及其在非IID数据上的有效性。
📝 Abstract
Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a hundredfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Update Masking
Federated Learning
Flat Minima
Regularization
Gradient Diversity
🔎 Similar Papers
No similar papers found.
