Communication Optimization Schemes for Accelerating Distributed Deep Learning Systems
Source abstract
Abstract
In a distributed deep learning system, a parameter server and workers must communicate to exchange gradients and parameters, and the communication cost increases as the number of workers increases. This paper presents a communication data optimization scheme to mitigate the decrease in throughput due to communication performance bottlenecks in distributed deep learning. To optimize communication, we propose two methods. The first is a layer dropping scheme to reduce communication data. The layer dropping scheme we propose compares the representative values of each hidden layer with a threshold value. Furthermore, to guarantee the training accuracy, we store the gradients that are not transmitted to the parameter server in the worker’s local cache. When the value of gradients stored in the worker’s local cache is greater than the threshold, the gradients stored in the worker’s local cache are transmitted to the parameter server. The second is an efficient threshold selection method. Our threshold selection method computes the threshold by replacing the gradients with the L1 norm of each hidden layer. Our data optimization scheme reduces the communication time by about 81% and the total training time by about 70% in a 56 Gbit network environment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Stochastic Gradient Optimization Techniques · Machine Learning and ELM
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Korea
Department of Computer and Information Security, Sejong University, Seoul 05006, Korea
Funding
National Research Foundation of Korea · Gyeonggi-do Regional Research Center · Institute for Information and Communications Technology Promotion · National Supercomputing Center of Korea
References (25)
Show source-provided references
1. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F.-F. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
3. Real, E., Aggarwal, A., Huang, Y., and Le, Q. (2018). Regularized Evolution for Image Classifier Architecture Search. arXiv.
4. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q.V., and Wu, Y. (2019, January 8–14). Gpipe: Efficient training of giant neural networks using pipeline parallelism. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
5. Zinkevich, M., Weimer, M., Li, L., and Smola, A.J. (2010, January 6–11). Parallelized Stochastic Gradient Descent. Proceedings of the Advances in Neural Information Processing Systems 23, Vancouver, BC, Canada.
6. Kim, Y., Choi, H., Lee, J., Kim, J., Jei, H., and Roh, H. (2019, January 16–20). Efficient Large-Scale Deep Learning Framework for Heterogeneous Multi-GPU Cluster. Proceedings of the 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W), Umea, Sweden.10.1109/FAS-W.2019.00050 ↗
7. Kim, Y., Lee, J., Kim, J., Jei, H., and Roh, H. (2018, January 3–7). Efficient Multi-GPU Memory Management for Deep Learning Acceleration. Proceedings of the 2018 IEEE 3rd International Workshops on Foundations and Applications of Self* Systems (FAS*W), Trento, Italy.10.1109/FAS-W.2018.00023 ↗
8. Kim, Y., Choi, H., Lee, J., Kim, J.S., Jei, H., and Roh, H. (2020). Towards an optimized distributed deep learning framework for a heterogeneous multi-GPU cluster. Clust. Comput., 23.10.1007/s10586-020-03144-9 ↗
9. Comprehensive techniques of multi-GPU memory optimization for deep learning acceleration10.1007/s10586-019-02974-6 ↗
10. Naumov, M., Kim, J., Mudigere, D., Sridharan, S., Wang, X., Zhao, W., Yilmaz, S., Kim, C., Yuen, H., and Ozdal, M. (2020). Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems. arXiv.
11. Heigold, G., McDermott, E., Vanhoucke, V., Senior, A., and Bacchiani, M. (2014, January 4–9). Asynchronous stochastic optimization for sequence training of deep neural networks. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854672 ↗
12. Zhang, H., Zheng, Z., Xu, S., Dai, W., Ho, Q., Liang, X., Hu, Z., Wei, J., Xie, P., and Xing, E.P. (2017, January 12–14). Poseidon: An efficient communication architecture for distributed deep learning on {GPU} clusters. Proceedings of the 2017 {USENIX} Annual Technical Conference ({USENIX}{ATC} 17), Santa Clara, CA, USA.
13. Sattler, F., Wiedemann, S., Müller, K., and Samek, W. (2019, January 14–19). Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.10.1109/IJCNN.2019.8852172 ↗
14. Dong, J., Cao, Z., Zhang, T., Ye, J., Wang, S., Feng, F., Zhao, L., Liu, X., Song, L., and Peng, L. (2020, January 22–26). EFLOPS: Algorithm and System Co-Design for a High Performance Distributed Training Platform. Proceedings of the 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), San Diego, CA, USA.10.1109/HPCA47549.2020.00056 ↗
15. Li, Y., Park, J., Alian, M., Yuan, Y., Qu, Z., Pan, P., Wang, R., Schwing, A., Esmaeilzadeh, H., and Kim, N.S. (2018, January 20–24). A Network-Centric Hardware/Algorithm Co-Design to Accelerate Distributed Training of Deep Neural Networks. Proceedings of the 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Fukuoka, Japan.10.1109/MICRO.2018.00023 ↗
16. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
17. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
18. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
19. Aji, A.F., and Heafield, K. (2017, January 7–11). Sparse Communication for Distributed Gradient Descent. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1045 ↗
20. McIlraith, S.A., and Weinberger, K.Q. (2018, January 2–7). AdaComp: Adaptive Residual Gradient Compression for Data-Parallel Distributed Training. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), The 30th Innovative Applications of Artificial Intelligence (IAAI-18), and The 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, LA, USA.
21. Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. (2017, January 4–9). TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. Proceedings of the 31st International Conference on Neural Information Processing Systems—NIPS’17, Long Beach, CA, USA.
22. Sergeev, A., and Balso, M.D. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv.
23. (2020, November 10). TensorFlow: An Open Source Machine Learning Library for Research and Production. Available online: https://www.tensorflow.org/.
24. Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W.J. (2017). Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv.
25. (2020, November 10). KISTI Neuron. Available online: https://www.ksc.re.kr/eng/resource/neuron.