Optimizing Single DGX-A100 System: Overcoming GPU Limitations via Efficient Parallelism and Scheduling for Large Language Models
Source abstract
Abstract
In this study, we introduce a novel training algorithm specifically designed to overcome the limitations of GPU memory on a single DGX-A100 system. By utilizing the CPU and main memory in the training process and applying a strategy of division and parallelization, our algorithm enhances the size of the trainable language model and the batch size. In addition, we developed a comprehensive management system to effectively manage the execution of the algorithm. This system systematically controls the training process and resource usage, while also enabling the asynchronous deployment of tasks. Finally, we proposed a scheduling technique integrated into the management system, promoting efficient task scheduling in a complex, heterogeneous training environment. These advancements equip researchers with the ability to work with larger models and batch sizes, even when faced with limited GPU memory.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Parallel Computing and Optimization Techniques · Topic Modeling · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering, Korea University, Seoul 02841, Republic of Korea
References (36)
Show source-provided references
1. Attention is all you need
2. Mars, M. (2022). From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough. Appl. Sci., 12.10.3390/app12178805 ↗
3. Garrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., and Ureña-López, L.A. (2021). A survey on bias in deep NLP. Appl. Sci., 11.10.20944/preprints202103.0049.v1 ↗
4. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2023, August 10). Improving language understanding by generative pre-training. Available online: https://www.mikecaptain.com/resources/pdf/GPT-1.pdf.
5. Language models are few-shot learners
6. Nvidia Corporation (2020). NVIDIA DGX A100|DATA SHEET, Nvidia Corporation.
7. Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. (2021, January 14–19). Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA.10.1145/3458817.3476205 ↗
8. Ren, J., Rajbhandari, S., Aminabadi, R.Y., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y. (2021, January 14–16). {ZeRO-Offload}: Democratizing {Billion-Scale} model training. Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC 21), Virtual.
9. Envpool: A highly parallel reinforcement learning environment execution engine
10. Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv.
11. Gupta, A., and Berant, J. (2020). Gmat: Global memory augmentation for transformers. arXiv.
12. Zero: Memory optimizations toward training trillion parameter models
13. Choi, H., and Lee, J. (2021). Efficient use of gpu memory for large-scale deep learning model training. Appl. Sci., 11.10.3390/app112110377 ↗
14. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
15. Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. (2018). Pipedream: Fast and efficient pipeline parallel dnn training. arXiv.10.1145/3341301.3359646 ↗
16. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
17. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q.V., and Wu, Y. (2019, January 8–14). Gpipe: Efficient training of giant neural networks using pipeline parallelism. Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS’19), Vancouver, BC, Canada.
18. Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D., and Stoica, I. (2021, January 18–24). Terapipe: Token-level pipeline parallelism for training large-scale language models. Proceedings of the International Conference on Machine Learning. PMLR, Online.
19. Bian, Z., Xu, Q., Wang, B., and You, Y. (2021). Maximizing parallelism in distributed training for huge neural networks. arXiv.
20. Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., and Young, C. (2018, January 13–16). Mesh-tensorflow: Deep learning for supercomputers. Proceedings of the 32rd International Conference on Neural Information Processing Systems (NIPS’18), Montréal, BC, Canada.
21. Song, L., Chen, F., Zhuo, Y., Qian, X., Li, H., and Chen, Y. (2020, January 22–26). AccPar: Tensor partitioning for heterogeneous deep learning accelerators. Proceedings of the 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), San Diego, CA, USA.10.1109/HPCA47549.2020.00036 ↗
22. A Survey on Auto-Parallelism of Large-Scale Deep Learning Training10.1109/TPDS.2023.3281931 ↗
23. Fu, Q., Chukka, R., Achorn, K., Atta-fosu, T., Canchi, D.R., Teng, Z., White, J., and Schmidt, D.C. (2022). Deep Learning Models on CPUs: A Methodology for Efficient Training. arXiv.
24. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., and Yang, K. (2012, January 3–6). Large scale distributed deep networks. Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS’12), New Orleans, LA, USA.
25. Zhang, H., Huang, H., and Han, H. (2021). A novel heterogeneous parallel convolution Bi-LSTM for speech emotion recognition. Appl. Sci., 11.10.20944/preprints202108.0433.v1 ↗
26. Shin, W., Yoo, K.H., and Baek, N. (2020). Large-Scale data computing performance comparisons on sycl heterogeneous parallel processing layer implementations. Appl. Sci., 10.10.3390/app10051656 ↗
27. Choukse, E., Sullivan, M.B., O’connor, M., Erez, M., Pool, J., Nellans, D., and Keckler, S.W. (June, January 30). Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs. Proceedings of the 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), Valencia, Spain.10.1109/ISCA45697.2020.00080 ↗
28. Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers10.14778/3551793.3551828 ↗
29. Zou, D., Jin, X., Yu, X., Zhang, H., and Demmel, J. (2023). Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping. arXiv.
30. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv.
31. HeterPS: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environments10.1016/j.future.2023.05.032 ↗
32. Jain, A., Moon, T., Benson, T., Subramoni, H., Jacobs, S.A., Panda, D.K., and Essen, B.V. (2021, January 17–21). SUPER: SUb-graph parallelism for TransformERs. Proceedings of the 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Portland, OR, USA.10.1109/IPDPS49936.2021.00071 ↗
33. Zinkevich, M., Weimer, M., Li, L., and Smola, A. (2010, January 6–9). Parallelized stochastic gradient descent. Proceedings of the 23th International Conference on Neural Information Processing Systems (NIPS’10), Vancouver, BC, Canada.
34. Kennedy, R.K., Khoshgoftaar, T.M., Villanustre, F., and Humphrey, T. (2019). A parallel and distributed stochastic gradient descent implementation using commodity clusters. J. Big Data, 6.10.1186/s40537-019-0179-2 ↗
35. RIDE: Real-time massive image processing platform on distributed environment10.1186/s13640-018-0279-5 ↗
36. Exploring the limits of transfer learning with a unified text-to-text transformer