A Case Study on Virtual HPC Container Clusters and Machine Learning Applications
Source abstract
Abstract
This article delves into the innovative application of Docker containers as High-Performance-Computing (HPC) environments, presenting the construction and operational efficiency of virtual container clusters. The study primarily focused on the integration of Docker technology in HPC, evaluating its feasibility and performance implications. A portion of the research was devoted to developing a virtual container cluster using Docker. Although the first Docker-enabled HPC studies date back several years, the approach remains highly relevant today, as modern AI-driven science demands portable, reproducible software stacks that can be deployed across heterogeneous, accelerator-rich clusters. Furthermore, the article explores the development of advanced distributed applications, with a special emphasis on Machine Learning (ML) algorithms. Key findings of the study include the successful implementation and operation of a Docker-based cluster. Additionally, the study successfully showcases a Python application using ML for anomaly detection in system logs, highlighting its effective execution in a virtual cluster. This research not only contributes to the understanding of Docker’s potential in distributed environments but also opens avenues for future explorations in the field of containerized HPC solutions and their applications in different areas.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Distributed and Parallel Computing Systems · Cloud Computing and Resource Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer and Control Engineering, Rzeszow University of Technology, Powstancow Warszawy 12, 35-959 Rzeszow, Poland
References (46)
Show source-provided references
1. Kononowicz, T., and Czarnul, P. (2022). Performance Assessment of Using Docker for Selected MPI Applications in a Parallel Environment Based on Commodity Hardware. Appl. Sci., 12.10.3390/app12168305 ↗
2. Anzt, H., Bienz, A., Luszczek, P., and Baboulin, M. (2022). Virtual Clusters: Isolated, Containerized HPC Environments in Kubernetes. High Performance Computing. ISC High Performance 2022 International Workshops, Springer.10.1007/978-3-031-23220-6 ↗
3. Zhou, N., Georgiou, Y., Pospieszny, M., Zhong, L., Zhou, H., Niethammer, C., Pejak, B., Marko, O., and Hoppe, D. (2021). Container orchestration on HPC systems through Kubernetes. J. Cloud Comput., 10.10.1186/s13677-021-00231-z ↗
4. Kuity, A., and Peddoju, S.K. (2023). Investigating performance metrics for container-based HPC environments using x86 and OpenPOWER systems. J. Cloud Comput., 12.10.1186/s13677-023-00546-z ↗
5. Rak, T., and Żyła, R. (2022). Using Data Mining Techniques for Detecting Dependencies in the Outcoming Data of a Web-Based System. Appl. Sci., 12.10.3390/app12126115 ↗
6. Kumar, M., and Kaur, G. (2022, January 23–24). Containerized MPI Application on InfiniBand-based HPC: An Empirical Study. Proceedings of the 2nd International Conference on Innovative Sustainable Computational Technologies (CISCT), Dehradun, India.
7. Lee, S., and Lee, J. (2024). Collective Communication Performance Evaluation for Distributed Deep Learning Training. Appl. Sci., 14.10.3390/app14125100 ↗
8. Mallick, P.K., Bhoi, A.K., Marques, G., and de Albuquerque, V.H.C. (2021). Own HPC Cluster Based on Virtual Operating System. Cognitive Informatics and Soft Computing, Springer.10.1007/978-981-16-1056-1 ↗
9. Jedynak, R. (2011, January 7–8). Practical Implementation of a Virtual Computer Cluster. Proceedings of the Present Day Trends of Innovations, Dubnica nad Váhom, Slovakia.
10. Sergeev, A., Rezedinova, E., and Khakhina, A. (2022). Stress testing of Docker containers running on a Windows operating system. J. Phys. Conf. Ser., 2339.10.1088/1742-6596/2339/1/012010 ↗
11. Leahy, D., and Thorpe, C. (2022, January 17–18). Zero Trust Container Architecture (ZTCA): A Framework for Applying Zero Trust Principals to Docker Containers. Proceedings of the 17th International Conference on Cyber Warfare and Security (ICCWS 2022), Albany, NY, USA.10.34190/iccws.17.1.35 ↗
12. Zhang, Y., Zhang, Y., Mao, X., Wu, Y., Lin, B., and Wang, S. (2022, January 15–18). Recommending Base Image for Docker Containers based on Deep Configuration Comprehension. Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA.10.1109/SANER53432.2022.00060 ↗
13. Kniep, C. (2025, June 03). Containerization of High Performance Compute Workloads Using Docker. Available online: http://web.archive.org/web/20160705235428/http://doc.qnib.org/2014-11-05_Whitepaper_Docker-MPI-workload.pdf.
14. Felter, W., Ferreira, A., Rajamony, R., and Rubio, J. (2015, January 29–31). An updated performance comparison of virtual machines and Linux containers. Proceedings of the 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Philadelphia, PA, USA.10.1109/ISPASS.2015.7095802 ↗
15. Chung, M.T., Quang-Hung, N., Nguyen, M.T., and Thoai, N. (2016, January 27–29). Using Docker in high performance computing applications. Proceedings of the 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE), Ha-Long, Vietnam.10.1109/CCE.2016.7562612 ↗
16. Ermakov, A., and Vasyukov, A. (2017). Testing Docker Performance for HPC Applications. arXiv.
17. Rak, T. (2023). Performance Evaluation of an API Stock Exchange Web System on Cloud Docker Containers. Appl. Sci., 13.10.3390/app13179896 ↗
18. Sobieraj, M., and Kotyński, D. (2024). Docker Performance Evaluation across Operating Systems. Appl. Sci., 14.10.3390/app14156672 ↗
19. Jacobsen, D. (2015, January 26–30). Contain This, Unleashing Docker for HPC. Proceedings of the Cray Users Group, Chicago, IL, USA.
20. Khalilnasl, H., Ferrari, P., Flammini, A., and Sisinni, E. (2025). On the Use of Containers for LoRaWAN Node Virtualization: Practice and Performance Evaluation. Electronics, 14.10.20944/preprints202502.1945.v1 ↗
21. Yu, H.E., and Huang, W. (2015). Building a Virtual HPC Cluster with Auto Scaling by the Docker. arXiv.
22. Nguyen, N., and Bein, D. (2017, January 9–11). Distributed MPI cluster with Docker Swarm mode. Proceedings of the 2017 IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC.2017.7868429 ↗
23. Torrez, A., Randles, T., and Priedhorsky, R. (2019, January 18). HPC Container Runtimes have Minimal or No Performance Impact. Proceedings of the 2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), Denver, CO, USA.10.1109/CANOPIE-HPC49598.2019.00010 ↗
24. Sparks, J. (2019). Enabling Docker for HPC. Concurr. Comput. Pract. Exp., 31.10.1002/cpe.5018 ↗
25. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. arXiv.
26. A Multi-Containerized Application using Docker Containers and Kubernetes Clusters
27. Roth, K., Pemula, L., Zepeda, J., Scholkopf, B., Brox, T., and Gehler, P. (2022, January 18–24). Towards Total Recall in Industrial Anomaly Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01392 ↗
28. Deng, A., and Hooi, B. (2021). Graph Neural Network-Based Anomaly Detection in Multivariate Time Series. arXiv.10.1609/aaai.v35i5.16523 ↗
29. Deng, H., and Li, X. (2022, January 18–24). Anomaly Detection via Reverse Distillation from One-Class Embedding. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00951 ↗
30. Li, C.L., Sohn, K., Yoon, J., and Pfister, T. (2021, January 20–25). CutPaste: Self-Supervised Learning for Anomaly Detection and Localization. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00954 ↗
31. Zavrtanik, V., Kristan, M., and Skocaj, D. (2021, January 10–17). DRAEM—A discriminatively trained reconstruction embedding for surface anomaly detection. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00822 ↗
32. Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv.
33. Snoek, J., Larochelle, H., and Adams, R.P. (2012, January 3–6). Practical Bayesian Optimization of Machine Learning Algorithms. Proceedings of the Neural Information Processing Systems, Lake Tahoe, NV, USA.
34. Ajasa, A.D., Chizari, H., and Alam, A. (2025). Database Security and Performance: A Case of SQL Injection Attacks Using Docker-Based Virtualisation and Its Effect on Performance. Future Internet, 17.10.3390/fi17040156 ↗
35. A Docker-based federated learning framework design and deployment for multi-modal data stream classification10.1007/s00607-023-01179-5 ↗
36. A container-based workflow for distributed training of deep learning algorithms in HPC clusters10.1007/s10586-022-03798-7 ↗
37. Advanced Computing: Best Practices for Automating Image Deployment Using Docker, Cloud, and HPC for AI/ML Applications
38. (2025, June 03). GitHub. Available online: https://github.com/trak2025zzz/hpc-docker.
39. Borowiec, M., Piszko, R., and Rak, T. (2023). Knowledge Extraction and Discovery about Web System Based on the Benchmark Application of Online Stock Trading System. Sensors, 23.10.3390/s23042274 ↗
40. Borowiec, M., and Rak, T. (2023). Advanced Examination of User Behavior Recognition via Log Dataset Analysis of Web Applications Using Data Mining Techniques. Electronics, 12.10.3390/electronics12214408 ↗
41. Long Short-term Memory10.1162/neco.1997.9.8.1735 ↗
42. Wei, Y., Jang-Jaccard, J., Xu, W., Sabrina, F., Camtepe, S., and Boulic, M. (2022). LSTM-Autoencoder based Anomaly Detection for Indoor Air Quality Time Series Data. arXiv.10.1109/JSEN.2022.3230361 ↗
43. Spam review detection using LSTM autoencoder: An unsupervised approach10.1007/s10660-020-09413-4 ↗
44. A Study on Tool Breakage Detection During Milling Process Using LSTM-Autoencoder and Gaussian Mixture Model10.1007/s12541-022-00647-w ↗
45. Sergeev, A., and Balso, M.D. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv.
46. Ranjan, C. (2025, June 03). Step-by-Step Understanding LSTM Autoencoder Layers. Available online: https://towardsdatascience.com/step-by-step-understanding-lstm-autoencoder-layers-ffab055b6352.