SimpleScale: Simplifying the Training of an LLM Model Using 1024 GPUs
Source abstract
Abstract
LLMs are trained using many thousands of GPUs in well-known conventional models. It is necessary to address numerous issues in the training process, such as manual data collection organization, data parallel, model parallel, evaluation, testing, deployment, transferring large data streams, detecting errors, ongoing maintenance, and project management. A team of dozens of engineers is required to handle system problems in the training process. Therefore, it is time-consuming and expensive to build an efficient and fault-tolerant system based on Kubernetes. This paper develops SimpleScale for building LLMs based on FSDP and Slurm, which is a simple and efficient training system that includes the training agent, the efficient parallel strategy, the optimal step of checkpoint, and so on. Using the proposed system enables us to significantly accelerate the process of building the LLM without incurring substantial time spent on various manual issues. The proposed 1024-GPU cluster was tested on TinyLlama, which has 1.1 billion parameters and 300 billion tokens. For example, utilizing a 16-H100 GPU cluster accelerated the traditional TinyLlama training time costs from 89.05 days to 39.05 days. In the future, the project team plans to integrate Flash-Attention3, aiming for an MFU of more than 60% while maintaining the acceleration achieved in the present work.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Data Storage Technologies · Cloud Computing and Resource Management · Parallel Computing and Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Jinan Institute of Supercomputing Technology, Jinan 251013, China
Department of Computer Science, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 251013, China
National Engineering Research Center for Visual Technology, School of Computer Science, Peking University, Beijing 100871, China
Funding
New 20 Project of Higher Education of Jinan
References (27)
Show source-provided references
1. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv.
2. ChatGPT10.1093/postmj/qgad056 ↗
3. BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. (2024). Llm2vec: Large language models are secretly powerful text encoders. arXiv.
4. Ben Zaken, E., Goldberg, Y., and Ravfogel, S. (2022, January 22–27). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.10.18653/v1/2022.acl-short.1 ↗
5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., and Phang, J. (2022). Gpt-neox-20b: An open-source autoregressive language model. arXiv.10.18653/v1/2022.bigscience-1.9 ↗
6. De la Rosa, J., and Fernández, A. (2022, January 20–25). Zero-shot Reading Comprehension and Reasoning for Spanish with BERTIN GPT-J-6B. Proceedings of the IberLEF@ SEPLN, A Coruña, Spain.
7. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training compute-optimal large language models. arXiv.
8. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2020, January 16–20). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online (Virtual Conference).10.18653/v1/2020.emnlp-demos.6 ↗
9. Zong, M., and Krishnamachari, B. (2022). A survey on GPT-3. arXiv.
10. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
11. Kubernetes
12. Jiang, Z., Lin, H., Zhong, Y., Huang, Q., Chen, Y., Zhang, Z., Peng, Y., Li, X., Xie, C., and Nong, S. (2024, January 17). {MegaScale}: Scaling large language model training to more than 10,000 {GPUs}. Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), Santa Clara, CA, USA.
13. Language models are few-shot learners
14. Li, H., Koto, F., Wu, M., Aji, A.F., and Baldwin, T. (2023). Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. arXiv.
15. Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv.
16. Flashattention-3: Fast and accurate attention with asynchrony and low-precision
17. Droste, A., Kuhn, M., and Ludwig, T. (2015, January 15). MPI-checker: Static analysis for MPI. Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, Austin, TX, USA.10.1145/2833157.2833159 ↗
18. Yoo, A.B., Jette, M.A., and Grondona, M. (2003, January 24). Slurm: Simple linux utility for resource management. Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, Seattle, WA, USA.10.1007/10968987_3 ↗
19. HPC cloud for scientific and business applications: Taxonomy, vision, and research challenges
20. Introduction to infiniband for end users
21. Awan, A.A., Chu, C.-H., Subramoni, H., and Panda, D.K. (2018, January 23–26). Optimized broadcast for deep learning workloads on dense-GPU InfiniBand clusters: MPI or NCCL?. Proceedings of the 25th European MPI Users’ Group Meeting, Barcelona, Spain.10.1145/3236367.3236381 ↗
22. Imambi, S., Prakash, K.B., and Kanagachidambaresan, G. (2021). PyTorch. Programming with TensorFlow: Solution for Edge Computing Applications, Springer.10.1007/978-3-030-57077-4 ↗
23. Ultra-performance Pascal GPU and NVLink interconnect10.1109/MM.2017.37 ↗
24. Pawlowski, B., Noveck, D., Robinson, D., and Thurlow, R. (2000, January 22–25). The NFS version 4 protocol. Proceedings of the 2nd International System Administration and Networking Conference (SANE 2000), Maastricht, The Netherlands.
25. Zhang, P., Zeng, G., Wang, T., and Lu, W. (2024). Tinyllama: An open-source small language model. arXiv.
26. AI, L. (2025, June 15). LitGPT. Available online: https://github.com/Lightning-AI/litgpt.
27. Nvidia hopper h100 gpu: Scaling performance10.1109/MM.2023.3256796 ↗