Low-Scalability Distributed Systems for Artificial Intelligence: A Comparative Study of Distributed Deep Learning Frameworks for Image Classification
Source abstract
Abstract
Artificial intelligence has experienced tremendous growth in various areas of knowledge, especially in computer science. Distributed computing has become necessary for storing, processing, and generating large amounts of information essential for training artificial intelligence models and algorithms that allow knowledge to be created from large amounts of data. Currently, cloud services offer products for running distributed data training, such as NVIDIA Deep Learning Solutions, Amazon SageMaker, Microsoft Azure, and Google Cloud AI Platform. These services have a cost that adapts to the needs of users who require high processing performance to perform their artificial intelligence tasks. This study highlights the relevance of distributed computing in image processing and classification tasks using a low-scalability distributed system built with devices considered obsolete. To this end, two of the most widely used libraries for the distributed training of deep learning models, PyTorch’s Distributed Data Parallel and Distributed TensorFlow, were implemented and evaluated using the ResNet50 model as a basis for image classification, and their performance was compared with modern environments such as Google Colab and a recent Workstation. The results demonstrate that even with low scalability and outdated distributed systems, comprehensive artificial intelligence tasks can still be performed, reducing investment time and costs. With the results obtained and experiments conducted in this study, we aim to promote technological sustainability through device recycling to facilitate access to high-performance computing in key areas such as research, industry, and education.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Scientific Computing and Data Management · Advanced Neural Network Applications · Parallel Computing and Optimization Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratorio de Invenciones Aplicadas a la Industria (LIAI), Unidad Académica de Ingeniería Eléctrica, Universidad Autónoma de Zacatecas, Zacatecas 98000, Mexico
Laboratorio de Sistemas Inteligentes de Visión Artificial, Posgrado en Ingeniería y Tecnología Aplicada, Universidad Autónoma de Zacatecas, Zacatecas 98000, Mexico
Unidad Académica de Ingeniería Eléctrica, Universidad Autónoma de Zacatecas, Zacatecas 98000, Mexico
References (54)
Show source-provided references
1. Machine learning and deep learning10.1007/s12525-021-00475-2 ↗
2. Artificial neural networks10.1145/846183.846192 ↗
3. Zhang, F., Petersen, M., Johnson, L., Hall, J., and O’Bryant, S.E. (2022). Hyperparameter tuning with high performance computing machine learning for imbalanced Alzheimer’s disease data. Appl. Sci., 12.10.3390/app12136670 ↗
4. Reasoning within fuzzy description logics10.1613/jair.813 ↗
5. Fault diagnosis in high-speed computing systems using big data analytics integrated evolutionary computing on the Internet of Everything platform
6. Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., and Gao, J. (2016, January 2–4). Deep reinforcement learning for dialogue generation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1127 ↗
7. An object-based convolutional neural network (OCNN) for urban land use classification10.1016/j.rse.2018.06.034 ↗
8. Lv, X., Ming, D., Lu, T., Zhou, K., Wang, M., and Bao, H. (2018). A new method for region-based majority voting CNNs for very high resolution image classification. Remote Sens., 10.10.3390/rs10121946 ↗
9. Towards an optimized distributed deep learning framework for a heterogeneous multi-GPU cluster10.1007/s10586-020-03144-9 ↗
10. HeterPS: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environments10.1016/j.future.2023.05.032 ↗
11. Sun, P., Feng, W., Han, R., Yan, S., and Wen, T. (2019). Optimizing network performance for distributed DNN training on GPU clusters: ImageNet/AlexNet training in 1.5 minutes. arXiv.10.1109/TBDATA.2019.2957478 ↗
12. High performance GPU computing based approaches for oil spill detection from multi-temporal remote sensing data10.1016/j.rse.2017.03.024 ↗
13. Breast histopathology with high-performance computing and deep learning10.31577/cai_2020_4_780 ↗
14. High performance computing for hyperspectral remote sensing10.1109/JSTARS.2010.2095495 ↗
15. High performance numerical computing for high energy physics: A new challenge for big data science10.1155/2014/507690 ↗
16. Bridging the science and technology by modern mathematical methods and high performance computing10.21136/AM.2021.0362-20 ↗
17. Select Your Own Counterparts: Self-Supervised Graph Contrastive Learning with Positive Sampling
18. Xu, P., Shi, S., and Chu, X. (2017, January 10–12). Performance Evaluation of Deep Learning Tools in Docker Containers. Proceedings of the 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM), Chengdu, China.10.1109/BIGCOM.2017.32 ↗
19. Bahrampour, S., Ramakrishnan, N., Schott, L., and Shah, M. (2015). Comparative Study of Deep Learning Software Frameworks. arXiv.
20. Comparative Study of Distributed Deep Learning Tools on Supercomputers
21. Hegde, V., and Usmani, S. (2016). Parallel and Distributed Deep Learning. arXiv.
22. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. Tech. Rep., Available online: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
23. Mohajerani, S., Krammer, T.A., and Saeedi, P. (2018, January 29–31). A Cloud Detection Algorithm for Remote Sensing Images Using Fully Convolutional Neural Networks. Proceedings of the 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), Vancouver, BC, Canada.10.1109/MMSP.2018.8547095 ↗
24. Mohajerani, S., and Saeedi, P. (2019). Cloud-Net: An End-to-End Cloud Detection Algorithm for Landsat 8 Imagery. arXiv.10.1109/IGARSS.2019.8898776 ↗
25. Gürbüz, S., and Aydin, G. (2022, January 21–23). Monkeypox Skin Lesion Detection Using Deep Learning Models. Proceedings of the 2022 International Conference on Computers and Artificial Intelligence Technologies (CAIT), Istanbul, Turkey.10.1109/CAIT56099.2022.10072140 ↗
26. Kshemkalyani, A.D., and Singhal, M. (2008). Distributed Computing: Principles, Algorithms, and Systems, Cambridge University Press. [1st ed.].10.1017/CBO9780511805318 ↗
27. A Review on Modern Distributed Computing Paradigms: Cloud Computing, Jungle Computing and Fog Computing10.2498/cit.1002381 ↗
28. Cisco Systems (2024, December 12). Cisco Fog Computing Solutions Unleash the Power of the Internet of Things. Available online: https://docplayer.net/20003565-Cisco-fog-computing-solutions-unleash-the-power-of-the-internet-of-things.html.
29. Merenda, M., Porcaro, C., and Iero, D. (2020). Edge Machine Learning for AI-Enabled IoT Devices: A Review. Sensors, 20.10.3390/s20092533 ↗
30. Sunny, R.T., and Thampi, S.M. (2012). Survey on Distributed Data Mining in P2P Networks. arXiv.
31. Misirli, J., and Casalicchio, E. (2024). An Analysis of Methods and Metrics for Task Scheduling in Fog Computing. Future Internet, 16.10.3390/fi16010016 ↗
32. G-Hadoop: MapReduce across distributed data centers for data-intensive computing10.1016/j.future.2012.09.001 ↗
33. Algorithmic Aspects of Parallel Data Processing10.1561/1900000055 ↗
34. Measuring the Effects of Data Parallelism on Neural Network Training
35. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv.
36. Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training Deep Nets with Sublinear Memory Cost. arXiv.
37. Jiang, Y., Zhu, Y., Lan, C., Yi, B., Cui, Y., and Guo, C. (2020, January 4–6). A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters. Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation (OSDI’20).
38. Sergeev, A., and Del Balso, M. (2018). Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv.
39. Gu, J., Chowdhury, M., Shin, K.G., Zhu, Y., Jeon, M., Qian, J., Liu, H., and Guo, C. (2019, January 26–28). Tiresias: A GPU Cluster Manager for Distributed Deep Learning. Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation (NSDI’19), Boston, MA, USA.
40. Efficient MPI-AllReduce for Large-Scale Deep Learning on GPU-Clusters10.1002/cpe.5574 ↗
41. NVIDIA (2025, April 10). NVIDIA Collective Communications Library (NCCL). Available online: https://docs.nvidia.com/deeplearning/nccl/index.html.
42. Optimized Large-Message Broadcast for Deep Learning Workloads: MPI, MPI+NCCL, or NCCL2?10.1016/j.parco.2019.03.005 ↗
43. Amdahl’s Law in the Multicore Era10.1109/MC.2008.209 ↗
44. How Amdahl’s Law Limits the Performance of Large Artificial Neural Networks10.1186/s40708-019-0097-2 ↗
45. TensorFlow (2025, April 10). Distributed Training with TensorFlow. Available online: https://www.tensorflow.org/guide/distributed_training?hl=es-419.
46. PyTorch (2025, April 10). Distributed Training Overview. Available online: https://pytorch.org/tutorials/beginner/dist_overview.html.
47. Google (2025, April 10). Colaboratory FAQ. Available online: https://research.google.com/colaboratory/intl/es/faq.html.
48. Google (2025, April 10). Administrar Google Colab para Organizaciones. Available online: https://support.google.com/a/answer/13177581.
49. NVIDIA Corporation (2025, March 21). GeForce GTX 1050 Ti. NVIDIA. Available online: https://www.nvidia.com/es-la/geforce/products/10series/geforce-gtx-1050/.
50. Intel Corporation (2025, March 21). Intel® Core™ i7-7700K Processor (8M Cache, up to 4.50 GHz). Intel ARK. Available online: https://ark.intel.com/products/97129.
51. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
52. (2025, May 25). Horovod: Distributed Training Framework for TensorFlow, Keras, PyTorch, and MXNet. Available online: https://horovod.readthedocs.io/en/latest/keras.html.
53. (2025, April 11). Available online: https://top500.org/.
54. VNDHR: Variational Single Nighttime Image Dehazing for Enhancing Visibility in Intelligent Transportation Systems via Hybrid Regularization