Check-QZP: A Lightweight Checkpoint Mechanism for Deep Learning Frameworks
Source abstract
Abstract
In deep learning (DL) frameworks, a checkpoint operation is widely used to store intermediate variable values (e.g., weights, biases, and gradients) on storage media. This operation helps to reduce the recovery time of running a machine learning (ML) model after sudden power failures or random crashes. However, the checkpoint operation can stall the overall training step of the running model and waste expensive hardware resources by leaving the GPU in idle sleep during the checkpoint operation. In addition, the completion time of the checkpoint operation is unpredictable in cloud server environments (e.g., AWS and Azure) because excessive I/O operations issued by other running applications interfere with the checkpoint operations in the storage stacks. To efficiently address the above two problems, we carefully designed Check-QZP, which reduces the amount of data required for checkpoint operations and parallelizes executions on the CPU and GPU by understanding the internal behaviors of the training step. For the evaluation, we implemented Check-QZP and compared it with the traditional approach in real-world multi-tenant scenarios. In the evaluation, Check-QZP outperformed the baseline in all cases in terms of the overall checkpoint time and the amount of data generated by the checkpoint operations, reducing them by up to 87.5% and 99.8%, respectively. In addition, Check-QZP achieved superior training speeds compared to the baseline.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Advanced Neural Network Applications · Radiation Effects in Electronics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, College of IT Convergence, Gachon University, Seongnam-si 13120, Republic of Korea
Department of Computer Engineering, Changwon National University, Changwon-si 51140, Republic of Korea
Funding
Gachon University · Korean government (MSIT)
References (55)
Show source-provided references
1. A Survey on Deep Learning: Algorithms, Techniques, and Applications
2. Object Detection in 20 Years: A Survey10.1109/JPROC.2023.3238524 ↗
3. Gwak, M., Cha, J., Yoon, H., Kang, D., and An, D. (2024). Lightweight Transformer Model for Mobile Application Classification. Sensors, 24.10.3390/s24020564 ↗
4. Kwon, C., and Kang, D. (2024). Overlay-ML: Unioning Memory and Storage Space for On-Device AI on Mobile Devices. Appl. Sci., 14.10.3390/app14073022 ↗
5. A Deep Learning Approach for Task Offloading in Multi-UAV Aided Mobile Edge Computing10.1109/ACCESS.2022.3208584 ↗
6. Mohan, J., Phanishayee, A., and Chidambaram, V. (2021, January 23–25). CheckFreq: Frequent, Fine-Grained DNN Checkpointing. Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST 21), Online.
7. Eisenman, A., Matam, K.K., Ingram, S., Mudigere, D., Krishnamoorthi, R., Nair, K., Smelyanskiy, M., and Annavaram, M. (2022, January 4–6). Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models. Proceedings of the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), Renton, WA, USA.
8. Multi-Tier GPU Virtualization for Deep Learning in Cloud-Edge Systems10.1109/TPDS.2023.3274957 ↗
9. Kang, D. (2024). Delay-D: Research on the Lifespan and Performance of Storage Devices in Unmanned Aerial Vehicles. Aerospace, 11.10.3390/aerospace11010047 ↗
10. Hassan, S.A., Rahim, T., and Shin, S.Y. (2019, January 16–18). Real-time UAV Detection based on Deep Learning Network. Proceedings of the 2019 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Republic of Korea.10.1109/ICTC46691.2019.8939564 ↗
11. Deep Learning Based Hand Gesture Recognition and UAV Flight Controls10.1007/s11633-019-1194-7 ↗
12. Autonomous Driving Architectures: Insights of Machine Learning and Deep Learning Algorithms
13. Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues10.1016/j.array.2021.100057 ↗
14. Shin, J., Piran, M.J., Song, H.K., and Moon, H. (2022, January 17). UAV-assisted and deep learning-driven object detection and tracking for autonomous driving. Proceedings of the 5th International ACM Mobicom Workshop on Drone Assisted Wireless Communications for 5G and Beyond, Sydney, Australia.10.1145/3555661.3560856 ↗
15. Zhang, S., Zhuo, L., Zhang, H., and Li, J. (2020). Object Tracking in Unmanned Aerial Vehicle Videos via Multifeature Discrimination and Instance-Aware Attention Network. Remote Sens., 12.10.3390/rs12162646 ↗
16. Wu, H.H., Zhou, Z., Feng, M., Yan, Y., Xu, H., and Qian, L. (2019, January 11–14). Real-Time Single Object Detection on The UAV. Proceedings of the 2019 International Conference on Unmanned Aircraft Systems (ICUAS), Atlanta, GA, USA.10.1109/ICUAS.2019.8797866 ↗
17. Recalibrating global data center energy-use estimates10.1126/science.aba3758 ↗
18. UAV-Aided Data Collection for Information Freshness in Wireless Sensor Networks10.1109/TWC.2020.3041750 ↗
19. Flight Time Minimization of UAV for Data Collection Over Wireless Sensor Networks10.1109/JSAC.2018.2864420 ↗
20. Jeon, M., Venkataraman, S., Qian, J., Phanishayee, A., Xiao, W., and Yang, F. (2024, September 13). Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications. Technical Report, Microsoft Research. Available online: https://www.microsoft.com/en-us/research/uploads/prod/2018/05/gpu_sched_tr.pdf.
21. Zobaed, S., Mokhtari, A., Champati, J.P., Kourouma, M., and Salehi, M.A. (2022, January 6–9). Edge-multiAI: Multi-tenancy of latency-sensitive deep learning applications on edge. Proceedings of the 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing (UCC), Vancouver, WA, USA.10.1109/UCC56403.2022.00012 ↗
22. (2021, July 20). PyTorch. Available online: https://pytorch.org/.
23. (2021, July 20). Tensorflow. Available online: https://www.tensorflow.org/?hl=en.
24. Chien, S.D., Markidis, S., Sishtla, C., Santos, L., Herman, P., Narasimhamurthy, S., and Laure, E. (2018, January 12). Characterizing Deep-Learning I/O Workloads in TensorFlow. Proceedings of the 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS), Dallas, TX, USA.10.1109/PDSW-DISCS.2018.00011 ↗
25. Park, S., and Bahn, H. (2023). Performance Analysis of Container Effect in Deep Learning Workloads and Implications. Appl. Sci., 13.10.3390/app132111654 ↗
26. Dey, T., Sato, K., Nicolae, B., Guo, J., Domke, J., Yu, W., Cappello, F., and Mohror, K. (2020, January 18–22). Optimizing Asynchronous Multi-Level Checkpoint/Restart Configurations with Machine Learning. Proceedings of the 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), New Orleans, LA, USA.10.1109/IPDPSW50202.2020.00174 ↗
27. Nicolae, B., Li, J., Wozniak, J.M., Bosilca, G., Dorier, M., and Cappello, F. (2020, January 11–14). DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models. Proceedings of the 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), Melbourne, Australia.10.1109/CCGrid49817.2020.00-76 ↗
28. Axboe, J. (2024, September 13). Flexible I/O Tester (FIO). Available online: https://github.com/axboe/fio.
29. Overlapped Data Processing Scheme for Accelerating Training and Validation in Machine Learning10.1109/ACCESS.2022.3189373 ↗
30. Deep learning10.1038/nature14539 ↗
31. Back-propagation neural networks for modeling complex systems10.1016/0954-1810(94)00011-S ↗
32. Li, M., Zhang, T., Chen, Y., and Smola, A.J. (2014, January 24–27). Efficient mini-batch training for stochastic optimization. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.10.1145/2623330.2623612 ↗
33. Gupta, T., Krishnan, S., Kumar, R., Vijeev, A., Gulavani, B., Kwatra, N., Ramjee, R., and Sivathanu, M. (2024, January 22–25). Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures. Proceedings of the Nineteenth European Conference on Computer Systems, Athens, Greece.10.1145/3627703.3650085 ↗
34. Xiang, L., Lu, X., Zhang, R., and Hu, Z. (2024, January 19–22). SSDC: A Scalable Sparse Differential Checkpoint for Large-scale Deep Recommendation Models. Proceedings of the 2024 IEEE International Symposium on Circuits and Systems (ISCAS), Singapore.10.1109/ISCAS58744.2024.10557880 ↗
35. Jang, H., Song, J., Jung, J., Park, J., Kim, Y., and Lee, J. (2024, January 2–6). Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System. Proceedings of the 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Edinburgh, UK.10.1109/HPCA57654.2024.00034 ↗
36. A deep learning scheme for efficient multimedia IoT data compression10.1016/j.adhoc.2022.102998 ↗
37. Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. (2021, January 14–19). ZeRO-infinity: Breaking the GPU memory wall for extreme scale deep learning. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA.10.1145/3458817.3476205 ↗
38. Zhang, W., Yu, S., Wang, L., Guo, W., and Leung, M.F. (2024). Constrained Symmetric Non-Negative Matrix Factorization with Deep Autoencoders for Community Detection. Mathematics, 12.10.3390/math12101554 ↗
39. Lv, C., Yang, L., Zhang, X., Li, X., Wang, P., and Du, Z. (2023). Unmanned Aerial Vehicle-Based Compressed Data Acquisition for Environmental Monitoring in WSNs. Sensors, 23.10.3390/s23208546 ↗
40. UAV-aided projection-based compressive data gathering in wireless sensor networks10.1109/JIOT.2018.2878834 ↗
41. Ebrahimi, D., Sharafeddine, S., Ho, P.H., and Assi, C. (2018, January 9–13). Data Collection in Wireless Sensor Networks Using UAV and Compressive Data Gathering. Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates.10.1109/GLOCOM.2018.8647924 ↗
42. Zheng, W., Song, Y., Guo, Z., Cui, Y., Gu, S., Mao, Y., and Cheng, L. (2019, January 24–26). Target-based Resource Allocation for Deep Learning Applications in a Multi-tenancy System. Proceedings of the 2019 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, USA.10.1109/HPEC.2019.8916403 ↗
43. Nikolaidis, S., Venieris, S.I., and Venieris, I.S. (2023, January 9–12). MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge. Proceedings of the 2023 IEEE Symposium on Computers and Communications (ISCC), Gammarth, Tunisia.10.1109/ISCC58397.2023.10217872 ↗
44. Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators10.1038/s41467-023-40770-4 ↗
45. Qlora: Efficient finetuning of quantized llms
46. Guo, M., Dong, Z., and Keutzer, K. (2023). SANA: Sensitivity-Aware Neural Architecture Adaptation for Uniform Quantization. Appl. Sci., 13.10.3390/app131810329 ↗
47. (2021, July 20). Gzip. Available online: https://www.gzip.org/.
48. (2023, July 20). Multiprocessing. Available online: https://docs.python.org/ko/3/library/multiprocessing.html.
49. Deep learning in multi-object detection and tracking: State of the art10.1007/s10489-021-02293-7 ↗
50. A survey of detection-based video multi-object tracking10.1016/j.displa.2022.102317 ↗
51. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
52. Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., and Wei, Y. (2022, January 23–27). MOTR: End-to-End Multiple-Object Tracking with TRansformer. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19812-0_38 ↗
53. Du, D., Zhu, P., Wen, L., Bian, X., Lin, H., Hu, Q., Peng, T., Zheng, J., Wang, X., and Zhang, Y. (2019, January 27–28). VisDrone-DET2019: The Vision Meets Drone Object Detection in Image Challenge Results. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Seoul, Republic of Korea.
54. Mokhtari, A., Hossen, M.A., Jamshidi, P., and Salehi, M.A. (2022, January 10–16). Felare: Fair scheduling of machine learning tasks on heterogeneous edge systems. Proceedings of the 2022 IEEE 15th International Conference on Cloud Computing (CLOUD), Barcelona, Spain.10.1109/CLOUD55607.2022.00069 ↗
55. Filho, C.P., Marques, E., Chang, V., Dos Santos, L., Bernardini, F., Pires, P.F., Ochi, L., and Delicato, F.C. (2022). A systematic literature review on distributed machine learning in edge computing. Sensors, 22.10.3390/s22072665 ↗