Lazy Aggregation for Heterogeneous Federated Learning
Source abstract
Abstract
Federated learning (FL) is a distributed neural network training paradigm with privacy protection. With the premise of ensuring that local data isn’t leaked, multi-device cooperation trains the model and improves its normalization. Unlike centralized training, FL is susceptible to heterogeneous data, biased gradient estimations hinder convergence of the global model, and traditional sampling techniques cannot apply FL due to privacy constraints. Therefore, this paper proposes a novel FL framework, federated lazy aggregation (FedLA), which reduces aggregation frequency to obtain high-quality gradients and improve robustness in non-IID. To judge the aggregating timings, the change rate of the models’ weight divergence (WDR) is introduced to FL. Furthermore, the collected gradients also facilitate FL walking out of the saddle point without extra communications. The cross-device momentum (CDM) mechanism could significantly improve the upper limit performance of the global model in non-IID. We evaluate the performance of several popular algorithms, including FedLA and FedLA with momentum (FedLAM). The results show that FedLAM achieves the best performance in most scenarios and the performance of the global model can also be improved in IID scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Stochastic Gradient Optimization Techniques · Mobile Crowdsensing and Crowdsourcing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, North China University of Technology, Beijing 100144, China
Information Security Center, State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
School of Information Engineering, Inner Mongolia University of Science and Technology, Baotou 014010, China
Funding
NSFC · Fundamental Research Funds for Beijing Municipal Commission of Education · Beijing Urban Governance Research Base of North China University of Technology · Natural Science Foundation of Inner Mongolia · Baotou Kundulun District Science and technology plan project · Inner Mongolia discipline inspection and supervision big data laboratory open project fund
References (33)
Show source-provided references
1. Federated learning: Challenges, methods, and future directions
2. Federated learning on non-IID data: A survey10.1016/j.neucom.2021.07.098 ↗
3. McMahan, H.B., Moore, E., Ramage, D., Hampson, S., and Arcas, B.A. (2017, January 20–22). Communication-Efficient Learning of Deep Networks from Decentralized Data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA.
4. Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. (2020, January 26–30). On the Convergence of FedAvg on Non-IID Data. Proceedings of the 8th International Conference on Learning Representations, Virtual.
5. Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv.
6. Zeng, S., Li, Z., Yu, H., He, Y., Xu, Z., Niyato, D., and Yu, H. (2022). Heterogeneous federated learning via grouped sequential-to-parallel training. International Conference on Database Systems for Advanced Applications, Springer.10.1007/978-3-031-00126-0_34 ↗
7. Federated optimization in heterogeneous networks
8. Shamir, O., Srebro, N., and Zhang, T. (2014, January 21–26). Communication-efficient distributed optimization using an approximate newton-type method. Proceedings of the 31th International Conference on Machine Learning, Beijing, China.
9. Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S.J., Stich, S.U., and Suresh, A.T. (2020, January 12–18). SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. Proceedings of the 37th on International Conference on Machine Learning, Virtual.
10. Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., and Saligrama, V. (2021, January 3–7). Federated learning based on dynamic regularization. Proceedings of the 9th International Conference on Learning Representations, Virtual.
11. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptationof deep networks. Proceedings of the 34th on International Conference on Machine Learning, Sydney, NSW, Australia.
12. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach
13. Arivazhagan, M.G., Aggarwal, V., Singh, A.K., and Choudhary, S. (2019). Federated learning with personalization layers. arXiv.
14. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraint10.1109/TNNLS.2020.3015958 ↗
15. An efficient framework for clustered federated learning
16. Multi-center federated learning: Clients clustering for better personalization
17. Duan, M., Liu, D., Ji, X., Liu, R., Liang, L., Chen, X., and Tan, Y. (2020). FedGroup: Efficient clustered federated learning via decomposed data-driven measure. arXiv.10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00042 ↗
18. Federated Learning with Soft Clustering10.1109/JIOT.2021.3113927 ↗
19. Yao, X., Huang, T., Zhang, R.-X., Li, R., and Sun, L. (2019). Federated learning with unbiased gradient aggregation and controllable meta updating. arXiv.
20. Tuor, T., Wang, S., Ko, B.J., Liu, C., and Leung, K.K. (2021, January 10–15). Overcoming noisy and irrelevant data in federated learning. Proceedings of the 25th International Conference on Pattern Recognition, Virtual.10.1109/ICPR48806.2021.9412599 ↗
21. The calculation of posterior distributions by data augmentation10.1080/01621459.1987.10478458 ↗
22. Duan, M., Liu, D., Chen, X., Tan, Y., Ren, J., Qiao, L., and Liang, L. (2019, January 17–20). Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. Proceedings of the 2019 IEEE 37th International Conference on Computer Design, Abu Dhabi, United Arab Emirates.10.1109/ICCD46524.2019.00038 ↗
23. Buciluǎ, C., Caruana, R., and Niculescu-Mizil, A. (2006, January 20–23). Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA.10.1145/1150402.1150464 ↗
24. Ensemble distillation for robust model fusion in federated learning
25. Konečný, J., McMahan, H.B., Ramage, D., and Richtárik, P. (2016). Federated optimization: Distributed machine learning for on-device intelligence. arXiv.
26. Khaled, A., Mishchenko, K., and Richtárik, P. (2020, January 26–28). Tighter theory for local SGD on identical and heterogeneous data. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistic, Parerlmo, Sicily, Italy.
27. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
28. Accelerating federated learning via momentum gradient descent10.1109/TPDS.2020.2975189 ↗
29. Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. (2020). Federated learning with matched averaging. arXiv.
30. Yann, L., Corest, C., and Burges, C.J.C. (2021, October 10). The MNIST Database of Handwritten Digits. Available online: http://yann.lecun.com/exdb/mnist.
31. Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017, January 14–19). EMNIST: Extending MNIST to handwritten letters. Proceedings of the 2017 International Joint Conference on Neural Networks, Anchorage, AK, USA.10.1109/IJCNN.2017.7966217 ↗
32. Doon, R., Rawat, T.K., and Gautam, S. (December, January 30). Cifar-10 classification using deep convolutional neural network. Proceedings of the 2018 IEEE Punecon, Pune, India.10.1109/PUNECON.2018.8745428 ↗
33. Wang, Z., Fan, X., Qi, J., Wen, C., Wang, C., and Yu, R. (2021). Federated learning with fair averaging. arXiv.10.24963/ijcai.2021/223 ↗