FedOpt: Towards Communication Efficiency and Privacy Preservation in Federated Learning
Source abstract
Abstract
Artificial Intelligence (AI) has been applied to solve various challenges of real-world problems in recent years. However, the emergence of new AI technologies has brought several problems, especially with regard to communication efficiency, security threats and privacy violations. Towards this end, Federated Learning (FL) has received widespread attention due to its ability to facilitate the collaborative training of local learning models without compromising the privacy of data. However, recent studies have shown that FL still consumes considerable amounts of communication resources. These communication resources are vital for updating the learning models. In addition, the privacy of data could still be compromised once sharing the parameters of the local learning models in order to update the global model. Towards this end, we propose a new approach, namely, Federated Optimisation (FedOpt) in order to promote communication efficiency and privacy preservation in FL. In order to implement FedOpt, we design a novel compression algorithm, namely, Sparse Compression Algorithm (SCA) for efficient communication, and then integrate the additively homomorphic encryption with differential privacy to prevent data from being leaked. Thus, the proposed FedOpt smoothly trade-offs communication efficiency and privacy preservation in order to adopt the learning task. The experimental results demonstrate that FedOpt outperforms the state-of-the-art FL approaches. In particular, we consider three different evaluation criteria; model accuracy, communication efficiency and computation overhead. Then, we compare the proposed FedOpt with the baseline configurations and the state-of-the-art approaches, i.e., Federated Averaging (FedAvg) and the paillier-encryption based privacy-preserving deep learning (PPDL) on all these three evaluation criteria. The experimental results show that FedOpt is able to converge within fewer training epochs and a smaller privacy budget.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Privacy-Preserving Technologies in Data · Cryptography and Data Security · Stochastic Gradient Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Nagoya Institute of Technology, Nagoya 466-8555, Aichi, Japan
Nagoya Institute of Technology (NITech) Frontier Institute, Nagoya 466-8555, Aichi, Japan
Faculty of Informatics, Zagazig University, Zagazig 44519, Egypt
References (44)
Show source-provided references
1. Müller, V.C., and Bostrom, N. (2016). Future progress in artificial intelligence: A survey of expert opinion. Fundamental Issues of Artificial Intelligence, Springer.10.1007/978-3-319-26485-1_33 ↗
2. Deep sparse representation classifier for facial recognition and detection system10.1016/j.patrec.2019.03.006 ↗
3. Reinforcement learning and deep learning based lateral control for autonomous driving [application notes]10.1109/MCI.2019.2901089 ↗
4. Bayesian Deep Learning Based Probabilistic Load Forecasting in Smart Grids10.1109/TII.2019.2942353 ↗
5. Approaches and challenges of privacy preserving search over encrypted data10.1016/j.is.2018.11.004 ↗
6. McMahan, B., and Ramage, D. (2017). Federated learning: Collaborative machine learning without centralized training data. Google Res. Blog, 3.
7. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., and Le, Q.V. (2012). Large scale distributed deep networks. Advances in Neural Information Processing Systems, The MIT Press.
8. Sattler, F., Wiedemann, S., Müller, K.R., and Samek, W. (2019). Robust and communication-efficient federated learning from non-iid data. IEEE Trans. Neural Netw. Learn. Syst.10.1109/TNNLS.2019.2944481 ↗
9. Bhagoji, A.N., Chakraborty, S., Mittal, P., and Calo, S. (2018). Analyzing federated learning through an adversarial lens. arXiv.
10. Nasr, M., Shokri, R., and Houmansadr, A. (2018). Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks. arXiv.
11. Hao, M., Li, H., Luo, X., Xu, G., Yang, H., and Liu, S. (2019). Efficient and privacy-enhanced federated learning for industrial artificial intelligence. IEEE Trans. Ind. Inform.10.1109/TII.2019.2945367 ↗
12. Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., and d’Oliveira, R.G. (2019). Advances and open problems in federated learning. arXiv.
13. DeepPAR and DeepDPA: Privacy-Preserving and Asynchronous Deep Learning for Industrial IoT10.1109/TII.2019.2941244 ↗
14. Tripathy, A., Wang, Y., and Ishwar, P. (2019, January 24–27). Privacy-preserving adversarial networks. Proceedings of the 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.10.1109/ALLERTON.2019.8919758 ↗
15. Calibrating noise to sensitivity in private data analysis
16. Aji, A.F., and Heafield, K. (2017). Sparse communication for distributed gradient descent. arXiv.10.18653/v1/D17-1045 ↗
17. Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W.J. (2017). Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv.
18. Cheng, Y., Wang, D., Zhou, P., and Zhang, T. (2017). A survey of model compression and acceleration for deep neural networks. arXiv.
19. Shokri, R., and Shmatikov, V. (2015, January 12–16). Privacy-preserving deep learning. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA.10.1145/2810103.2813687 ↗
20. Peralta, G., Cid-Fuentes, R.G., Bilbao, J., and Crespo, P.M. (2019). Homomorphic Encryption and Network Coding in IoT Architectures: Advantages and Future Challenges. Electronics, 8.10.3390/electronics8080827 ↗
21. Vadhan, S. (2017). The complexity of differential privacy. Tutorials on the Foundations of Cryptography, Springer.10.1007/978-3-319-57048-8_7 ↗
22. Roy, P. (2018). A homomorphism based zero knowledge proof of authentication for chinese remainder theorem based secret sharing. Annual Convention of the Computer Society of India, Springer.10.1007/978-981-13-1343-1_47 ↗
23. McMahan, H.B., Moore, E., Ramage, D., and Hampson, S. (2016). Communication-efficient learning of deep networks from decentralized data. arXiv.
24. Privacy-preserving deep learning via additively homomorphic encryption
25. Fazio, N., Gennaro, R., Jafarikhah, T., and Skeith, W.E. (2017). Homomorphic secret sharing from paillier encryption. International Conference on Provable Security, Springer.10.1007/978-3-319-68637-0_23 ↗
26. Chilimbi, T., Suzue, Y., Apacible, J., and Kalyanaraman, K. (2014, January 6–8). Project adam: Building an efficient and scalable deep learning training system. Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), Broomfield, CO, USA.
27. Zinkevich, M., Weimer, M., Li, L., and Smola, A.J. (2010). Parallelized stochastic gradient descent. Advances in Neural Information Processing Systems, The MIT Press.
28. Li, M., Andersen, D.G., Smola, A.J., and Yu, K. (2014). Communication efficient distributed machine learning with the parameter server. Advances in Neural Information Processing Systems, The MIT Press.10.1145/2640087.2644155 ↗
29. Aono, Y., Hayashi, T., Wang, L., and Moriai, S. (2017). Privacy-preserving deep learning: Revisited and enhanced. International Conference on Applications and Techniques in Information Security, Springer.
30. Agarwal, N., Suresh, A.T., Yu, F.X.X., Kumar, S., and McMahan, B. (2018). cpSGD: Communication-efficient and differentially-private distributed SGD. Advances in Neural Information Processing Systems, Neural Information Processing Systems Conference.
31. Deep learning based communication over the air
32. Nishio, T., and Yonetani, R. (2019, January 20–24). Client selection for federated learning with heterogeneous resources in mobile edge. Proceedings of the 2019 IEEE International Conference on Communications (ICC 2019), Shanghai, China.10.1109/ICC.2019.8761315 ↗
33. Konečný, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., and Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv.
34. Bernstein, J., Wang, Y.X., Azizzadenesheli, K., and Anandkumar, A. (2018). signSGD: Compressed optimisation for non-convex problems. arXiv.
35. Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., and Qi, H. (May, January 29). Beyond inferring class representatives: User-level privacy leakage from federated learning. Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2019), Paris, France.10.1109/INFOCOM.2019.8737416 ↗
36. Privacy-preserving classification on deep neural network
37. Ma, X., Ma, J., Li, H., Jiang, Q., and Gao, S. (2018). PDLM: Privacy-preserving deep learning model on cloud with multiple keys. IEEE Trans. Serv. Comput.
38. Moriai, S. (2019, January 10–12). Privacy-Preserving Deep Learning via Additively Homomorphic Encryption. Proceedings of the 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), Kyoto, Japan.10.1109/ARITH.2019.00047 ↗
39. Mohassel, P., and Zhang, Y. (2017, January 22–26). Secureml: A system for scalable privacy-preserving machine learning. Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.10.1109/SP.2017.12 ↗
40. Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016, January 24–28). Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria.10.1145/2976749.2978318 ↗
41. Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H.B., Patel, S., Ramage, D., Segal, A., and Seth, K. (November, January 30). Practical secure aggregation for privacy-preserving machine learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3133982 ↗
42. Liu, Y., Ma, Z., Liu, X., Ma, S., Nepal, S., and Deng, R. (2019). Boosting privately: Privacy-preserving federated extreme boosting for mobile crowdsensing. arXiv.10.1109/ICDCS47774.2020.00017 ↗
43. Hao, M., Li, H., Xu, G., Liu, S., and Yang, H. (2019, January 20–24). Towards Efficient and Privacy-Preserving Federated Deep Learning. Proceedings of the ICC 2019-2019 IEEE International Conference on Communications (ICC), Shanghai, China.10.1109/ICC.2019.8761267 ↗
44. Privacy-preserving collaborative deep learning with unreliable participants10.1109/TIFS.2019.2939713 ↗