FedRDS: Federated Learning on Non-IID Data via Regularization and Data Sharing
Source abstract
Abstract
Federated learning (FL) is an emerging decentralized machine learning framework enabling private global model training by collaboratively leveraging local client data without transferring it centrally. Unlike traditional distributed optimization, FL trains the model at the local client and then aggregates it at the server. While this approach reduces communication costs, the local datasets of different clients are non-Independent and Identically Distributed (non-IID), which may make the local model inconsistent. The present study suggests a FL algorithm that leverages regularization and data sharing (FedRDS). The local loss function is adapted by introducing a regularization term in each round of training so that the local model will gradually move closer to the global model. However, when the client data distribution gap becomes large, adding regularization items will increase the degree of client drift. Based on this, we used a data-sharing method in which a portion of server data is taken out as a shared dataset during the initialization. We then evenly distributed these data to each client to mitigate the problem of client drift by reducing the difference in client data distribution. Analysis of experimental outcomes indicates that FedRDS surpasses some known FL methods in various image classification tasks, enhancing both communication efficacy and accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Mobile Crowdsensing and Crowdsourcing · Stochastic Gradient Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Engineering, Yunnan University, Kunming 650091, China
Key Lab of Intelligent Systems and Computing of Yunnan Province, Yunnan University, Kunming 650091, China
School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China
School of Electrical and Automation Engineering, Nanjing Normal University, Nanjing 210024, China
Funding
National Natural Science Foundation of China · Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology · Yunnan High-Level Talent Training Support Plan: Young Top Talent Special Project
References (42)
Show source-provided references
1. Federated Machine Learning: Concept and Applications10.1145/3298981 ↗
2. Federated Learning: Challenges, Methods, and Future Directions
3. McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), Ft. Lauderdale, FL, USA.
4. Federated optimization in heterogeneous networks
5. Smith, V., Chiang, C., Sanjabi, M., and Talwalkar, A. (2017, January 4–9). Federated Multi-Task Learning. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA.
6. Multi-task federated learning for personalised deep neural networks in edge computing10.1109/TPDS.2021.3098467 ↗
7. Fallah, A., Mokhtari, A., and Ozdaglar, A.E. (2020, January 6–12). Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, Online.
8. Personalized Federated Learning on Non-IID Data via Group-Based Meta-Learning
9. Hiding in the Crowd: Federated Data Augmentation for On-Device Learning10.1109/MIS.2020.3028613 ↗
10. Towards personalized federated learning10.1109/TNNLS.2022.3160699 ↗
11. Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2019, January 6–9). Federated Learning with Non-IID Data. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
12. DReS-FL: Dropout-Resilient Secure Federated Learning for Non-IID Clients via Secret Data Sharing
13. Korkmaz, C., Kocas, H.E., Uysal, A., Masry, A., Ozkasap, O., and Akgun, B. (2020, January 2–5). Chain fl: Decentralized federated machine learning via blockchain. Proceedings of the 2020 Second International Conference on Blockchain Computing and Applications (BCCA), Antalya, Turkey.10.1109/BCCA50787.2020.9274451 ↗
14. Federated Learning for Cybersecurity: Concepts, Challenges, and Future Directions10.1109/TII.2021.3119038 ↗
15. Guest Editorial: Federated Learning for Industrial IoT in Industry 4.010.1109/TII.2021.3086509 ↗
16. Federated learning for predicting clinical outcomes in patients with COVID-1910.1038/s41591-021-01506-3 ↗
17. Gao, L., Fu, H., Li, L., Chen, Y., Xu, M., and Xu, C. (2022, January 18–24). FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00987 ↗
18. Horváth, S., Laskaridis, S., Almeida, M., Leontiadis, I., Venieris, S.I., and Lane, N.D. (2021, January 6–14). FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout. Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, Online.
19. Zhang, Y., Duan, L., and Cheung, N. (2022, January 20–23). Accelerating Federated Learning on Non-IID Data Against Stragglers. Proceedings of the IEEE International Conference on Sensing, Communication, and Networking, Online.10.1109/SECONWorkshops56311.2022.9926402 ↗
20. Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. (2021, January 3–7). FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. Proceedings of the 9th International Conference on Learning Representations, Online.
21. Li, Q., He, B., and Song, D. (2021, January 20–25). Model-contrastive federated learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01057 ↗
22. Privacy Threat and Defense for Federated Learning with Non-i.i.d. Data in AIoT10.1109/TII.2021.3073925 ↗
23. Li, T., Hu, S., Beirami, A., and Smith, V. (2020, January 13–18). Ditto: Fair and Robust Federated Learning through Personalization. Proceedings of the International Conference on Machine Learning, Online.
24. Entropy to Mitigate Non-IID Data Problem on Federated Learning for the Edge Intelligence Environment10.1109/ACCESS.2023.3298704 ↗
25. Dinh, C.T., Tran, N.H., and Nguyen, T.D. (2020). Personalized Federated Learning with Moreau Envelopes. arXiv.
26. Bezdek, J.C., and Hathaway, R.J. (2002, January 3–6). Some Notes on Alternating Optimization. Proceedings of the Advances in Soft Computing—AFSS 2002: 2002 AFSS International Conference on Fuzzy Systems, Calcutta, India.10.1007/3-540-45631-7_39 ↗
27. Nishio, T., and Yonetani, R. (2019, January 11–13). Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge. Proceedings of the 2019 IEEE International Conference on Communications, Changchun, China.10.1109/ICC.2019.8761315 ↗
28. Sun, Y., Zhou, S., and Gündüz, D. (2019, January 7–11). Energy-Aware Analog Aggregation for Federated Learning with Redundant Data. Proceedings of the ICC 2020—IEEE International Conference on Communications (ICC), Online.10.1109/ICC40277.2020.9148853 ↗
29. Toward Energy-Efficient Federated Learning Over 5G+ Mobile Devices10.1109/MWC.003.2100028 ↗
30. EPPDA: An Efficient Privacy-Preserving Data Aggregation Federated Learning Scheme10.1109/TNSE.2022.3153519 ↗
31. Personalized federated learning framework for network traffic anomaly detection10.1016/j.comnet.2022.108906 ↗
32. Optimizing Federated Learning with Deep Reinforcement Learning for Digital Twin Empowered Industrial IoT10.1109/TII.2022.3183465 ↗
33. FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning
34. Towards efficient scheduling of federated mobile devices under computational and statistical heterogeneity10.1109/TPDS.2020.3023905 ↗
35. A Survey on Collaborative Learning for Intelligent Autonomous Systems10.1145/3625544 ↗
36. Long, M., Wang, J., Ding, G., Sun, J., and Yu, P.S. (2013, January 1–8). Transfer Feature Learning with Joint Distribution Adaptation. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia.10.1109/ICCV.2013.274 ↗
37. Tan, S., Peng, X., and Saenko, K. (2020, January 13–19). Generalized Domain Adaptation with Covariate and Label Shift CO-ALignment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.
38. Cao, Z., Long, M., Wang, J., and Jordan, M.I. (2018, January 18–22). Partial Transfer Learning with Selective Adversarial Networks. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00288 ↗
39. Cao, K., Wei, C., Gaidon, A., Aréchiga, N., and Ma, T. (2019, January 8–14). Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Vancouver, BC, Canada.
40. Hayat, M., Khan, S., Zamir, S.W., Shen, J., and Shao, L. (November, January 27). Gaussian Affinity for Max-Margin Class Imbalanced Learning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00657 ↗
41. Jamal, M.A., Brown, M., Yang, M., Wang, L., and Gong, B. (2020, January 13–19). Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition From a Domain Adaptation Perspective. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00763 ↗
42. Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., and Saligrama, V. (2021, January 3–7). Federated Learning Based on Dynamic Regularization. Proceedings of the 9th International Conference on Learning Representations, Online.