An Intelligent Approach to Resource Allocation on Heterogeneous Cloud Infrastructures
Source abstract
Abstract
Cloud computing systems are rapidly evolving toward multicloud architectures supported on heterogeneous hardware. Cloud service providers are widely offering different types of storage infrastructures and multi-NUMA architecture servers. Existing cloud resource allocation solutions do not comprehensively consider this heterogeneous infrastructure. In this study, we present a novel approach comprised of a hierarchical framework based on genetic programming to solve problems related to data placement and virtual machine allocation for analytics applications running on heterogeneous hardware with a variety of storage types and nonuniform memory access. Our approach optimizes data placement using the Hadoop File System on heterogeneous storage devices on multicloud systems. It guarantees the efficient allocation of virtual machines on physical machines with multiple NUMA (nonuniform memory access) domains by minimizing contention between workloads. We prove that our solutions for data placement and virtual machine allocation outperform other state-of-the-art approaches.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Cloud Computing and Resource Management · IoT and Edge/Fog Computing · Advanced Data Storage Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Automatics and Electronics Department, Universidad Autónoma de Occidente, Cali 760030, Colombia
References (43)
Show source-provided references
1. Rydning, D.R.J.G.J. (2018). The Digitization of the World from Edge to Core, International Data Corporation.
2. Li, H., Li, H., Wen, Z., Mo, J., and Wu, J. (2017, January 15–17). Distributed heterogeneous storage based on data value. Proceedings of the 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China.10.1109/ITNEC.2017.8284985 ↗
3. Bezerra, A., Hernandez, P., Espinosa, A., and Moure, J.C. (2014, January 22–26). Job scheduling in Hadoop with Shared Input Policy and RAMDISK. Proceedings of the 2014 IEEE International Conference on Cluster Computing, CLUSTER 2014, Madrid, Spain.10.1109/CLUSTER.2014.6968788 ↗
4. Level Aware Data Placement Technique for Hybrid NAND Flash Storage of Log-Structured Merge-Tree Based Key-Value Store System10.1109/ACCESS.2020.3031322 ↗
5. Learning to schedule multi-NUMA virtual machines via reinforcement learning10.1016/j.patcog.2021.108254 ↗
6. Sheng, S., Chen, P., Chen, Z., Wu, L., and Yao, Y. (2021). Deep Reinforcement Learning-Based Task Scheduling in IoT Edge Computing. Sensors, 21.10.3390/s21051666 ↗
7. Hierarchical multi-agent optimization for resource allocation in cloud computing10.1109/TPDS.2020.3030920 ↗
8. Apache Software Foundation (2009). Hadoop, Apache Software Foundation.
9. Marquez, J., Gonzalez, J., and Mondragon, O.H. (2019, January 25–30). Heterogeneity-aware data placement in Hybrid Clouds. Proceedings of the 2019 International Conference on Cloud Computing (CLOUD 2019), San Diego, CA, USA.10.1007/978-3-030-23502-4_13 ↗
10. White, T. (2012). Hadoop: The Definitive Guide, O’Reilly.
11. Hadoop (2009). Archival Storage, SSD & Memory, Apache Software Foundation.
12. Blagodurov, S., Fedorova, A., Zhuravlev, S., and Kamali, A. (2010, January 11–15). A case for NUMA-aware contention management on multicore systems. Proceedings of the 2010 19th International Conference on Parallel Architectures and Compilation Techniques (PACT), Vienna, Austria.10.1145/1854273.1854350 ↗
13. A simplex method for function minimization10.1093/comjnl/7.4.308 ↗
14. Nonlinear optimization using the generalized reduced gradient method
15. The generalized reduced gradient method: A reliable tool for optimal design10.1115/1.3439249 ↗
16. Mitchell, M. (1998). An Introduction to Genetic Algorithms, MIT Press.
17. Mondragon, O.H., Bridges, P.G., Levy, S., Ferreira, K.B., and Widener, P. (2016, January 13–18). Understanding performance interference in next-generation HPC systems. Proceedings of the SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, USA.10.1109/SC.2016.32 ↗
18. Efficient resource allocation for on-demand mobile-edge cloud computing10.1109/TVT.2018.2846232 ↗
19. Optimizing VM allocation and data placement for data-intensive applications in cloud using ACO metaheuristic algorithm
20. Multi-objective communication-aware optimization for virtual machine placement in cloud datacenters
21. Razali, N.M., and Geraghty, J. (2011, January 6–8). Genetic algorithm performance with different selection strategies in solving TSP. Proceedings of the World Congress on Engineering, Hong Kong, China.
22. Gwiazda, T.D. (2021, June 26). Genetic Algorithms Reference Vol. I. Crossover for Single-Objective Numerical Optimization Problems [Online]. TOMASZGWIAZDA E-BOOKS. Available online: http://www.tomaszgwiazda.com/Genetic_algorithms_reference_first_40_pages.pdf.
23. A survey of evolutionary computation for resource management of processing in cloud computing10.1109/MCI.2015.2405351 ↗
24. Xen and the art of virtualization10.1145/1165389.945462 ↗
25. Duplyakin, D., Ricci, R., Maricq, A., Wong, G., Duerig, J., Eide, E., Stoller, L., Hibler, M., Johnson, D., and Webb, K. (2019, January 10–12). The Design and Operation of CloudLab. Proceedings of the USENIX Annual Technical Conference (ATC), Renton, WA, USA.
26. Luszczek, P., Dongarra, J.J., Koester, D., Rabenseifner, R., Lucas, B., Kepner, J., McCalpin, J., Bailey, D., and Takahashi, D. (2005). Introduction to the HPC Challenge Benchmark Suite, Ernest Orlando Lawrence Berkeley National Laboratory. Technical Report.10.1145/1188455.1188677 ↗
27. Project, X. (2015). Xen 4.2 Automatic NUMA Placement, Xen Project.
28. Voron, G., Thomas, G., Quéma, V., and Sens, P. (2017, January 23–26). An interface to implement NUMA policies in the Xen hypervisor. Proceedings of the Twelfth European Conference on Computer Systems, Belgrade, Serbia.10.1145/3064176.3064196 ↗
29. Keahey, K., Anderson, J., Zhen, Z., Riteau, P., Ruth, P., Stanzione, D., Cevik, M., Colleran, J., Gunawi, H.S., and Hammock, C. (2020, January 15–17). Lessons Learned from the Chameleon Testbed. Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20), Online.
30. Qian, J., Li, J., Ma, R., and Guan, H. (2018, January 7–10). Optimizing Virtual Resource Management for Consolidated NUMA Systems. Proceedings of the 2018 IEEE 36th International Conference on Computer Design (ICCD), Orlando, FL, USA.10.1109/ICCD.2018.00092 ↗
31. Wu, S., Sun, H., Zhou, L., Gan, Q., and Jin, H. (2016, January 12–16). vprobe: Scheduling virtual machines on numa systems. Proceedings of the 2016 IEEE International Conference on Cluster Computing (CLUSTER), Taipei, Taiwan.10.1109/CLUSTER.2016.60 ↗
32. Mondragon, O.H., Bridges, P.G., and Jones, T. (2015, January 16). Quantifying scheduling challenges for exascale system software. Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers, Portland, OR, USA.10.1145/2768405.2768413 ↗
33. Rao, J., Wang, K., Zhou, X., and Xu, C.Z. (2013, January 23–27). Optimizing virtual machine scheduling in NUMA multicore systems. Proceedings of the 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA), Shenzhen, China.
34. Ali, I.M., Sallam, K.M., Moustafa, N., Chakraborty, R., Ryan, M.J., and Choo, K.K.R. (2020). An automated task scheduling model using non-dominated sorting genetic algorithm ii for fog-cloud systems. IEEE Trans. Cloud Comput.
35. Tan, B., Ma, H., Mei, Y., and Zhang, M. (2020). A Cooperative Coevolution Genetic Programming Hyper-Heuristic Approach for On-line Resource Allocation in Container-based Clouds. IEEE Trans. Cloud Comput.10.1109/CEC.2019.8790220 ↗
36. Belgacem, A., Beghdad-Bey, K., and Nacer, H. (2020). Dynamic resource allocation method based on Symbiotic Organism Search algorithm in cloud computing. IEEE Trans. Cloud Comput.
37. Wang, W., Tornatore, M., Zhao, Y., Chen, H., Li, Y., Gupta, A., Zhang, J., and Mukherjee, B. (2021). Infrastructure-efficient Virtual-Machine Placement and Workload Assignment in Cooperative Edge-Cloud Computing over Backhaul Networks. IEEE Trans. Cloud Comput.
38. Liu, N., Li, Z., Xu, J., Xu, Z., Lin, S., Qiu, Q., Tang, J., and Wang, Y. (2017, January 5–8). A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA.10.1109/ICDCS.2017.123 ↗
39. A heterogeneous cloud storage platform with uniform data distribution by software-defined storage technologies10.1109/ACCESS.2019.2946962 ↗
40. HDFS Optimization Strategy Based On Hierarchical Storage of Hot and Cold Data10.1016/j.procir.2019.04.086 ↗
41. Wang, X., Zhang, J., and Shi, Y. (2020, January 19–23). Resource and Job Execution Context-Aware Hadoop Configuration Tuning. Proceedings of the 2020 IEEE 13th International Conference on Cloud Computing (CLOUD), Beijing, China.10.1109/CLOUD49709.2020.00029 ↗
42. Qian, Z., Gao, Y., Ji, M., Peng, H., Chen, P., Jin, Y., and Lu, S. (2019, January 16–18). Workload-Aware Scheduling for Data Analytics upon Heterogeneous Storage. Proceedings of the 2019 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), Xiamen, China.10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00088 ↗
43. Multi-objective optimization for virtual machine allocation and replica placement in virtualized hadoop10.1109/TPDS.2018.2837743 ↗