Secure Smart Communication Efficiency in Federated Learning: Achievements and Challenges
Source abstract
Abstract
Federated learning (FL) is known to perform machine learning tasks in a distributed manner. Over the years, this has become an emerging technology, especially with various data protection and privacy policies being imposed. FL allows for performing machine learning tasks while adhering to these challenges. As with the emergence of any new technology, there will be challenges and benefits. A challenge that exists in FL is the communication costs: as FL takes place in a distributed environment where devices connected over the network have to constantly share their updates, this can create a communication bottleneck. This paper presents the state-of-the-art of the conducted works on communication constraints of FL while maintaining the secure and smart properties that federated learning is known for. Overall, current challenges and possible methods for enhancing the FL models’ efficiency with a perspective on communication are discussed. This paper aims to bridge the gap in all conducted review papers by solely focusing on communication aspects in FL environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Cryptography and Data Security · Privacy, Security, and Data Protection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technology, Kennesaw State University, Marietta, GA 30060, USA
Department of Software Engineering and Game Development, Kennesaw State University, Marietta, GA 30060, USA
Department of Computing, Macquarie University, Sydney, NSW 2109, Australia
Department of Math and Computer Science, Brandon University, Brandon, MB R7A 6A9, Canada
Research Center for Interneural Computing, China Medical University, Taichung 40402, Taiwan
Department of Computer Science and Mathematics, Lebanese American University, Beirut 1102, Lebanon
Funding
U.S. SunTrust Fellow in Cybersecurity/Information Security Research Funding Program
References (94)
Show source-provided references
1. A review of machine learning for the optimization of production processes10.1007/s00170-019-03988-5 ↗
2. Pazzani, M. (1997, January 14–17). Comprehensible knowledge discovery: Gaining insight from data. Proceedings of the First Federal Data Mining Conference and Exposition, Citeseer, Newport Beach, CA, USA.
3. A machine learning approach to improving dynamic decision making10.1287/isre.2014.0513 ↗
4. Machine learning with big data: Challenges and approaches10.1109/ACCESS.2017.2696365 ↗
5. How the GDPR will change the world10.21552/EDPL/2016/3/4 ↗
6. McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B.A. (2017, January 9–11). Communication-efficient learning of deep networks from decentralized data. Proceedings of the Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA.
7. Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., and Zhou, Y. (2019, January 7–11). A hybrid approach to privacy-preserving federated learning. Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, Los Angeles, CA, USA.10.1145/3338501.3357370 ↗
8. Federated learning: A survey on enabling technologies, protocols, and applications10.1109/ACCESS.2020.3013541 ↗
9. Decentralized Authentication of Distributed Patients in Hospital Networks Using Blockchain10.1109/JBHI.2020.2969648 ↗
10. Consumer data privacy in a networked world: A framework for protecting privacy and promoting innovation in the global digital economy
11. The future of digital health with federated learning10.1038/s41746-020-00323-1 ↗
12. Privacy-preserving Traffic Flow Prediction: A Federated Learning Approach10.1109/JIOT.2020.2991401 ↗
13. Federated learning for drone authentication10.1016/j.adhoc.2021.102574 ↗
14. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges10.1109/TCOMM.2020.2990686 ↗
15. Zhao, Y., Zhao, J., Jiang, L., Tan, R., and Niyato, D. (2019). Mobile edge computing, blockchain and reputation-based crowdsourcing iot federated learning: A secure, decentralized and privacy-preserving system. arXiv.
16. Federated Learning-based Anomaly Detection for IoT Security Attacks10.1109/JIOT.2021.3077803 ↗
17. An Ensemble Multi-View Federated Learning Intrusion Detection for IoT10.1109/ACCESS.2021.3107337 ↗
18. An Ensemble of Deep Recurrent Neural Networks for Detecting IoT Cyber Attacks Using Network Traffic10.1109/JIOT.2020.2996425 ↗
19. Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage, D. (2018). Federated learning for mobile keyboard prediction. arXiv.
20. Chen, M., Mathews, R., Ouyang, T., and Beaufays, F. (2019). Federated learning of out-of-vocabulary words. arXiv.
21. Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., and Beaufays, F. (2018). Applied federated learning: Improving google keyboard query suggestions. arXiv.
22. Ramaswamy, S., Mathews, R., Rao, K., and Beaufays, F. (2019). Federated learning for emoji prediction in a mobile keyboard. arXiv.
23. Tian, L. (2020, November 13). Federated Learning: Challenges, Methods and Future Directions. Available online: https://blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-methods-and-future-directions/.
24. A survey on security and privacy of federated learning10.1016/j.future.2020.10.007 ↗
25. Canetti, R., Feige, U., Goldreich, O., and Naor, M. (1996, January 22–24). Adaptively secure multi-party computation. Proceedings of the Twenty-Eighth Annual ACM Symposium on Theory of Computing, Philadelphia, PA, USA.10.1145/237814.238015 ↗
26. Dwork, C. (2008, January 25–29). Differential privacy: A survey of results. Proceedings of the International Conference on Theory and Applications of Models of Computation, Xi’an, China.
27. Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P. (2017). Ensemble adversarial training: Attacks and defenses. arXiv.
28. Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., and Cummings, R. (2019). Advances and open problems in federated learning. arXiv.
29. Federated machine learning: Concept and applications10.1145/3339474 ↗
30. Federated learning in mobile edge networks: A comprehensive survey10.1109/COMST.2020.2986024 ↗
31. Połap, D., Srivastava, G., Lin, J.C.W., and Woźniak, M. (2021, January 20–24). Federated Learning Model with Augmentation and Samples Exchange Mechanism. Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland.10.1007/978-3-030-87986-0_19 ↗
32. Agent architecture of an intelligent medical system based on federated learning and blockchain technology
33. Reliable customer analysis using federated learning and exploring deep-attention edge intelligence10.1016/j.future.2021.08.028 ↗
34. Xu, R., Baracaldo, N., Zhou, Y., Anwar, A., and Ludwig, H. (2019, January 15). Hybridalpha: An efficient approach for privacy-preserving federated learning. Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, London, UK.10.1145/3338501.3357371 ↗
35. Reliable federated learning for mobile networks10.1109/MWC.001.1900119 ↗
36. From federated learning to federated neural architecture search: A survey10.1007/s40747-020-00247-z ↗
37. Tian, Z., Zhang, R., Hou, X., Liu, J., and Ren, K. (2020). FederBoost: Private Federated Learning for GBDT. arXiv.
38. Fedhealth: A federated transfer learning framework for wearable healthcare10.1109/MIS.2020.2988604 ↗
39. Federated learning for healthcare informatics10.1007/s41666-020-00082-4 ↗
40. A Privacy-Preserving Federated Learning for Multiparty Data Sharing in Social IoTs10.1109/TNSE.2021.3074185 ↗
41. A review of applications in federated learning10.1016/j.cie.2020.106854 ↗
42. Federated learning of predictive models from federated electronic health records10.1016/j.ijmedinf.2018.01.007 ↗
43. Nasajpour, M., Karakaya, M., Pouriyeh, S., and Parizi, R.M. (April, January 26). Federated Transfer Learning For Diabetic Retinopathy Detection Using CNN Architectures. Proceedings of the SoutheastCon 2022, IEEE, Mobile, AL, USA.10.1109/SoutheastCon48659.2022.9764031 ↗
44. Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., and He, B. (2021). A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Trans. Knowl. Data Eng., early access.
45. Federated Learning in a Medical Context: A Systematic Literature Review10.1145/3412357 ↗
46. Lyu, L., Yu, H., and Yang, Q. (2020). Threats to federated learning: A survey. arXiv.10.1007/978-3-030-63076-8_1 ↗
47. Challenges, applications and design aspects of federated learning: A survey10.1109/ACCESS.2021.3111118 ↗
48. Federated learning: Challenges, methods, and future directions
49. Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv.
50. Design and Analysis of Uplink and Downlink Communications for Federated Learning10.1109/JSAC.2020.3041388 ↗
51. Luping, W., Wei, W., and Bo, L. (2019, January 7–10). CMFL: Mitigating communication overhead for federated learning. Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), Dallas, TX, USA.
52. Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation10.1109/TNNLS.2019.2953131 ↗
53. Communication-efficient edge AI: Algorithms and systems10.1109/COMST.2020.3007787 ↗
54. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. (2018). Federated optimization in heterogeneous networks. arXiv.
55. Robust and communication-efficient federated learning from non-iid data10.1109/TNNLS.2019.2944481 ↗
56. Zhang, S., Choromanska, A.E., and LeCun, Y. (2015, January 7–12). Deep learning with elastic averaging SGD. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
57. Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv.
58. CoCoA: A general framework for communication-efficient distributed optimization
59. Briggs, C., Fan, Z., and Andras, P. (2020). Federated learning with hierarchical clustering of local updates to improve training on non-IID data. arXiv.10.1109/IJCNN48605.2020.9207469 ↗
60. K-means with Three different Distance Metrics
61. Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. (2020, January 26–28). Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. Proceedings of the International Conference on Artificial Intelligence and Statistics, PMLR, Online.
62. Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A.T. (2020, January 13–18). Scaffold: Stochastic controlled averaging for federated learning. Proceedings of the International Conference on Machine Learning, PMLR, Virtual.
63. Shamir, O., Srebro, N., and Zhang, T. (2014, January 22–24). Communication-efficient distributed optimization using an approximate newton-type method. Proceedings of the International Conference on Machine Learning, PMLR, Bejing, China.
64. Reddi, S.J., Konečnỳ, J., Richtárik, P., Póczós, B., and Smola, A. (2016). AIDE: Fast and communication efficient distributed optimization. arXiv.
65. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smithy, V. (2019, January 3–6). Feddane: A federated newton-type method. Proceedings of the 2019 IEEE 53rd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.10.1109/IEEECONF44664.2019.9049023 ↗
66. FedMCCS: Multicriteria client selection model for optimal IoT federated learning10.1109/JIOT.2020.3028742 ↗
67. Client selection and bandwidth allocation in wireless federated learning networks: A long-term perspective10.1109/TWC.2020.3031503 ↗
68. Nishio, T., and Yonetani, R. (2019, January 20–24). Client selection for federated learning with heterogeneous resources in mobile edge. Proceedings of the ICC 2019-2019 IEEE International Conference on Communications (ICC), Shanghai, China.10.1109/ICC.2019.8761315 ↗
69. Efficient training management for mobile crowd-machine learning: A deep reinforcement learning approach10.1109/LWC.2019.2917133 ↗
70. Cho, Y.J., Wang, J., and Joshi, G. (2020). Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies. arXiv.
71. Secure and Optimized Load Balancing for Multitier IoT and Edge-Cloud Computing Systems10.1109/JIOT.2020.3042433 ↗
72. Kamp, M., Adilova, L., Sicking, J., Hüger, F., Schlicht, P., Wirtz, T., and Wrobel, S. (2018, January 10–14). Efficient decentralized deep learning by dynamic model averaging. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Dublin, Ireland.10.1007/978-3-030-10925-7_24 ↗
73. Bui, T.D., Nguyen, C.V., Swaroop, S., and Turner, R.E. (2018). Partitioned variational inference: A unified framework encompassing federated and continual learning. arXiv.
74. Smith, V., Chiang, C.K., Sanjabi, M., and Talwalkar, A. (2017). Federated multi-task learning. arXiv.
75. Guha, N., Talwalkar, A., and Smith, V. (2019). One-shot federated learning. arXiv.
76. Fast-convergent federated learning10.1109/JSAC.2020.3036952 ↗
77. Tang, H., Gan, S., Zhang, C., Zhang, T., and Liu, J. (2018). Communication compression for decentralized training. arXiv.
78. Reisizadeh, A., Taheri, H., Mokhtari, A., Hassani, H., and Pedarsani, R. (2019). Robust and communication-efficient collaborative learning. arXiv.
79. He, C., Tan, C., Tang, H., Qiu, S., and Liu, J. (2019). Central server free federated learning over single-sided trust social networks. arXiv.
80. Lalitha, A., Kilinc, O.C., Javidi, T., and Koushanfar, F. (2019). Peer-to-peer federated learning on graphs. arXiv.
81. Roy, A.G., Siddiqui, S., Pölsterl, S., Navab, N., and Wachinger, C. (2019). Braintorrent: A peer-to-peer environment for decentralized federated learning. arXiv.
82. Rothchild, D., Panda, A., Ullah, E., Ivkin, N., Stoica, I., Braverman, V., Gonzalez, J., and Arora, R. (2020, January 13–18). Fetchsgd: Communication-efficient federated learning with sketching. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event.
83. Spring, R., Kyrillidis, A., Mohan, V., and Shrivastava, A. (2019). Compressing gradient optimizers via count-sketches. arXiv.
84. Li, S., Qi, Q., Wang, J., Sun, H., Li, Y., and Yu, F.R. (2020, January 7–11). GGS: General Gradient Sparsification for Federated Learning in Edge Computing. Proceedings of the ICC 2020-2020 IEEE International Conference on Communications (ICC), Virtual.10.1109/ICC40277.2020.9148987 ↗
85. Hu, R., Gong, Y., and Guo, Y. (2020). Sparsified Privacy-Masking for Communication-Efficient and Privacy-Preserving Federated Learning. arXiv.10.1109/ICC40277.2020.9149207 ↗
86. Sattler, F., Wiedemann, S., Müller, K.R., and Samek, W. (2019, January 14–19). Sparse binary compression: Towards distributed deep learning with minimal communication. Proceedings of the 2019 IEEE International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.10.1109/IJCNN.2019.8852172 ↗
87. Nicholls, J. (2021, April 28). Quantization in Deep Learning. Available online: https://medium.com/@joel_34050/quantization-in-deep-learning-478417eab72b/.
88. A linear speedup analysis of distributed deep learning with sparse and quantized communication
89. Amiri, M.M., Gunduz, D., Kulkarni, S.R., and Poor, H.V. (2020). Federated learning with quantized global model updates. arXiv.
90. Koloskova, A., Lin, T., Stich, S.U., and Jaggi, M. (2019). Decentralized deep learning with arbitrary communication compression. arXiv.
91. Dai, X., Yan, X., Zhou, K., Yang, H., Ng, K.K., Cheng, J., and Fan, Y. (2019). Hyper-sphere quantization: Communication-efficient sgd for federated learning. arXiv.
92. Shlezinger, N., Chen, M., Eldar, Y.C., Poor, H.V., and Cui, S. (2020, January 4–8). Federated learning with quantization constraints. Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054168 ↗
93. UVeQFed: Universal vector quantization for federated learning10.1109/TSP.2020.3046971 ↗
94. Liu, L., Zhang, J., Song, S., and Letaief, K.B. (2021). Hierarchical Quantized Federated Learning: Convergence Analysis and System Design. arXiv.