FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

2026-07-04

AI Summary

This work addresses the challenge of data heterogeneity in federated learning, which induces inconsistent trust across coordinate dimensions when using the AdamW optimizer, thereby impairing model convergence and performance. To mitigate this issue, the authors propose FedACT, a novel approach that introduces, for the first time in federated adaptive optimization, a coordinate-level trust scoring mechanism. FedACT dynamically modulates the magnitude of parameter updates by integrating global correction directions with local gradient information, enabling fine-grained and consistent adaptation across coordinates. Experimental results demonstrate that FedACT consistently outperforms existing methods across diverse architectures—including federated Vision Transformers, CNNs, and large language models—during both pretraining and fine-tuning, with particularly pronounced gains under highly heterogeneous data distributions.

Abstract

Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federated AdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM fine-tuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released.

Problem

• federated learning
• data heterogeneity
• adaptive optimization
• coordinate trust mismatch
• Transformer training

Innovation

• Federated Learning
• Adaptive Optimization
• Coordinate Trust Modulation
• Data Heterogeneity
• Transformer Training