A Survey on Malleability Solutions for High-Performance Distributed Computing
Source abstract
Abstract
Maintaining a high rate of productivity, in terms of completed jobs per unit of time, in High-Performance Computing (HPC) facilities is a cornerstone in the next generation of exascale supercomputers. Process malleability is presented as a straightforward mechanism to address that issue. Nowadays, the vast majority of HPC facilities are intended for distributed-memory applications based on the Message Passing (MP) paradigm. For this reason, many efforts are based on the Message Passing Interface (MPI), the de facto standard programming model. Malleability aims to rescale executions on-the-fly, in other words, reconfigure the number and layout of processes in running applications. Process malleability involves resources reallocation within the HPC system, handling processes of the application, and redistributing data among those processes to resume the execution. This manuscript compiles how different frameworks address process malleability, their main features, their integration in resource management systems, and how they may be used in user codes. This paper is a detailed state-of-the-art devised as an entry point for researchers who are interested in process malleability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Distributed and Parallel Computing Systems · Cloud Computing and Resource Management · Distributed systems and fault tolerance
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department de Ingeniería y Ciencia de Computadores, Universitat Jaume I, 12006 Castelló, Spain
Funding
MCIN/AEI/10.13039/501100011033 · Universitat Jaume I · postdoctoral fellowship · Valencian Region Government and European Social Funds
References (77)
Show source-provided references
1. Padua, D. (2011). TOP500. Encyclopedia of Parallel Computing, Springer.10.1007/978-0-387-09766-4 ↗
2. Balaprakash, P., Buntinas, D., Chan, A., Guha, A., Gupta, R., Narayanan, S.H.K., Chien, A.A., Hovland, P., and Norris, B. (2013, January 21–23). Exascale workload characterization and architecture implications. Proceedings of the 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Austin, TX, USA.10.1109/ISPASS.2013.6557153 ↗
3. Fault tolerance of MPI applications in exascale systems: The ULFM solution10.1016/j.future.2020.01.026 ↗
4. A survey of MPI usage in the US exascale computing project10.1002/cpe.4851 ↗
5. Toward Convergence in Job Schedulers for Parallel Supercomputers10.1007/BFb0022284 ↗
6. SRS: A framework for developing malleable and migratable applications for distributed systems
7. Plank, J.S., Beck, M., Elwasif, W.R., Moore, T., Swany, M., and Wolski, R. (1999, January 14–19). The Internet Backplane Protocol: Storage in the Network. Proceedings of the Network Storage Symposium, Portland, OR, USA.
8. El Maghraoui, K., Szymanski, B.K., and Varela, C. (2006, January 14–18). An Architecture for Reconfigurable Iterative MPI Applications in Dynamic Environments. Proceedings of the International Conference on Parallel Processing and Applied Mathematics, Columbus, OH, USA.10.1007/11752578_32 ↗
9. Plank, J.S., Beck, M., Kingsley, G., and Li, K. (1995, January 16–20). Libckpt: Transparent Checkpointing under Unix. Proceedings of the Usenix Winter Technical Conference, Orleans, LA, USA.
10. Desell, T., El Maghraoui, K., and Varela, C. (2004, January 5–8). Load Balancing of Autonomous Actors over Dynamic Networks. Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS’04), Big Island, HI, USA.
11. El Maghraoui, K., Desell, T.J., Szymanski, B.K., and Varela, C.A. (2007, January 14–17). Dynamic Malleability in Iterative MPI Applications. Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid), Rio de Janeiro, Brazil.10.1109/CCGRID.2007.45 ↗
12. Malleable Iterative MPI Applications
13. Moody, A., Bronevetsky, G., Mohror, K., and de Supinski, B.R. (2010, January 13–19). Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System. Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC10), Washington, DC, USA.10.1109/SC.2010.18 ↗
14. Lemarinier, P., Hasanov, K., Venugopal, S., and Katrinis, K. (2016, January 25–28). Architecting Malleable MPI Applications for Priority-driven Adaptive Scheduling. Proceedings of the 23rd European MPI Users’ Group Meeting (EuroMPI), Edinburgh, UK.10.1145/2966884.2966907 ↗
15. Bland, W., Bosilca, G., Bouteiller, A., Hérault, T., and Dongarra, J. (2022, April 13). A Proposal for User-Level Failure Mitigation in the MPI-3 Standard. Available online: https://www.semanticscholar.org/paper/A-Proposal-for-User-Level-Failure-Mitigation-in-the-Bland-Bosilca/3e697ef781bbce707e3729185a062fb3f580309e.
16. Träff, J.L., Benkner, S., and Dongarra, J.J. (2012). An Evaluation of User-Level Failure Mitigation Support in MPI. Recent Advances in the Message Passing Interface, Springer.10.1007/978-3-642-33518-1 ↗
17. Post-failure Recovery of MPI Communication Capability: Design and Rationale10.1177/1094342013488238 ↗
18. Kranzlmüller, D., Volkert, J., Kacsuk, P., and Dongarra, J. (2002). MPICH2: A New Start for MPI Implementations. Recent Advances in Parallel Virtual Machine and Message Passing Interface, Springer.10.1007/3-540-45825-5 ↗
19. Comprés, I., Mo-Hellenbrand, A., Gerndt, M., and Bungartz, H.J. (2016, January 25–28). Infrastructure and API Extensions for Elastic Execution of MPI Applications. Proceedings of the 23rd European MPI Users’ Group Meeting, Edinburgh, UK.10.1145/2966884.2966917 ↗
20. Feitelson, D., Rudolph, L., and Schwiegelshohn, U. (2003). SLURM: Simple Linux Utility for Resource Management. Job Scheduling Strategies for Parallel Processing, Springer.10.1007/10968987 ↗
21. Chadha, M. (2019). Adaptive Resource-Aware Batch Scheduling for HPC Systems. [Master’s Thesis, Technische Universität München].
22. Chadha, M., John, J., and Gerndt, M. (2022, April 13). Extending SLURM for Dynamic Resource-Aware Adaptive Batch Scheduling, Available online: http://xxx.lanl.gov/abs/2009.08289.
23. Sudarsan, R., and Ribbens, C.J. (2007, January 10–14). ReSHAPE: A Framework for Dynamic Resizing and Scheduling of Homogeneous Applications in a Parallel Environment. Proceedings of the International Conference on Parallel Processing, Xi’an, China.10.1109/ICPP.2007.73 ↗
24. Tadepalli, S. (2003). GEMS: A Fault Tolerant Grid Job Management System. [Master’s Thesis, Virginia Polytechnic Institute].
25. Swaminathan, G. (2004). A Scheduling Framework for Dynamically Resizable Parallel Applications. [Ph.D. Thesis, Virginia Tech].
26. Dongarra, J.J., and Whaley, R.C. (1997). A User’s Guide to the BLACS v1.1, University of Tennessee. Technical Report 94.
27. Blackford, L., Choi, J., and Cleary, A. (1997, January 13–15). ScaLAPACK: A linear algebra library for message-passing computers. Proceedings of the SIAM Conference, Stanford, CA, USA.10.1145/369028.369038 ↗
28. Sudarsan, R., and Ribbens, C. (2009, January 23–29). Scheduling Resizable Parallel Applications. Proceedings of the International Symposium on Parallel & Distributed Processing, Rome, Italy.10.1109/IPDPS.2009.5161077 ↗
29. Sudarsan, R., Ribbens, C.J., and Farkas, D. (2009, January 25–27). Dynamic Resizing of Parallel Scientific Simulations: A Case Study Using LAMMPS. Proceedings of the International Conference on Computational Science (ICCS), Baton Rouge, LA, USA.10.1007/978-3-642-01970-8_18 ↗
30. Enhancing the Performance of Malleable MPI Applications by Using Performance-aware Dynamic Reconfiguration10.1016/j.parco.2015.04.003 ↗
31. Mucci, P., Moore, S., Deane, C., and Ho, G. (1999, January 7–10). PAPI: A Portable Interface to Hardware Performance Counters. Proceedings of the Department of Defense HPCMP Users Group Conference, Monterey, CA, USA.
32. Radcliffe, N., Watson, L., and Sosonkina, M. (2011, January 24–26). A Comparison of Alternatives for Communicating with Spawned Processes. Proceedings of the 49th Annual Southeast Regional Conference, Kennesaw, GA, USA.10.1145/2016039.2016078 ↗
33. Martín, G., Marinescu, M.C., Singh, D.E., and Carretero, J. (2013, January 26–30). FLEX-MPI: An MPI Extension for Supporting Dynamic Load Balancing on Heterogeneous Non-dedicated Systems. Proceedings of the Euro-Par Parallel Processing, Aachen, Germany.10.1007/978-3-642-40047-6_16 ↗
34. Dongarra, J. (2022, March 03). Overview of PVM and MPI. Available online: http://www.netlib.org/utk/people/JackDongarra/pdf/pvm-mpi.pdf.
35. Jajaga, E., and Kllobocishta, J. (2012, January 12–15). MPI Parallel Implementation of Jacobi. Proceedings of the ICT Innovations, Ohrid, Republic of Macedonia. Available online: https://www.researchgate.net/publication/255719047_MPI_Parallel_Implementation_of_Jacobi.
36. Leveraging social networks for understanding the evolution of epidemics
37. Carretero, J., Garcia Blas, J., Singh, D., Isaila, F., Fahringer, T., Prodan, R., Bosilca, G., Lastovetsky, A., Symeonidou, C., and Pérez-Sánchez, H. (2014, January 9–12). Optimizations to enhance sustainability of MPI applications. Proceedings of the 21st European MPI Users’ Group Meeting, Kyoto, Japan.10.1145/2642769.2642797 ↗
38. Iserte, S. (2018). High-Throughput Computation through Efficient Resource Management. [Ph.D. Thesis, Universitat Jaume I].
39. Jeannot, E., Namyst, R., and Roman, J. (2011). Productive Cluster Programming with OmpSs. Euro-Par 2011 Parallel Processing: 17th International Conference, Euro-Par 2011, Bordeaux, France, 29 August–2 September 2011, Proceedings, Part I, Springer.10.1007/978-3-642-23397-5 ↗
40. DMR API: Improving cluster productivity by turning applications into malleable10.1016/j.parco.2018.07.006 ↗
41. Sainz, F., Bellon, J., Beltran, V., and Labarta, J. (2015, January 16–19). Collective Offload for Heterogeneous Clusters. Proceedings of the 2015 IEEE 22nd International Conference on High Performance Computing (HiPC), Washington, DC, USA.10.1109/HiPC.2015.20 ↗
42. Iserte, S., Mayo, R., Quintana-Ortí, E.S., Beltran, V., and Peña, A.J. (2017, January 14–17). Efficient Scalable Computing through Flexible Applications and Adaptive Workloads. Proceedings of the 46th International Conference on Parallel Processing Workshops (ICPPW), Bristol, UK.10.1109/ICPPW.2017.36 ↗
43. DMRlib: Easy-coding and Efficient Resource Management for Job Malleability10.1109/TC.2020.3022933 ↗
44. From NBODY1 to NBODY6: The Growth of an Industry*10.1086/316455 ↗
45. Dynamic reconfiguration of noniterative scientific applications10.1177/1094342018802347 ↗
46. An study of the effect of process malleability in the energy efficiency on GPU-based clusters10.1007/s11227-019-03034-x ↗
47. Prabhakaran, S., Iqbal, M., Rinke, S., Windisch, C., and Wolf, F. (2014, January 9–12). A Batch System with Fair Scheduling for Evolving Applications. Proceedings of the 2014 43rd International Conference on Parallel Processing, Minneapolis, MN, USA.10.1109/ICPP.2014.44 ↗
48. Prabhakaran, S., Neumann, M., Rinke, S., Wolf, F., Gupta, A., and Kale, L.V. (2015, January 25–29). A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications. Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium, Hyderabad, India.10.1109/IPDPS.2015.34 ↗
49. Bhandarkar, M., Kale, L.V., de Sturler, E., and Hoeflinger, J. (2001, January 28–30). Object-Based Adaptive Load Balancing for MPI Programs. Proceedings of the International Conference on Computational Science, San Francisco, CA, USA. LNCS 2074.10.1007/3-540-45718-6_13 ↗
50. Huang, C., Lawlor, O., and Kalé, L.V. (2003, January 2–4). Adaptive MPI. Proceedings of the 16th International Workshop on Languages and Compilers for Parallel Computing (LCPC 2003), College Station, TX, USA. LNCS 2958.
51. Kale, L., Bhandarkar, M., Jagathesan, N., Krishnan, S., and Yelon, J. (1996, January 12–16). Converse: An interoperable framework for parallel programming. Proceedings of the International Conference on Parallel Processing, Bloomingdale, IL, USA.
52. Rolim, J. (2000). Run-time support for adaptive load balancing. Parallel and Distributed Processing–15 IPDPS 2000 Workshops, Proceedings, Springer.10.1007/3-540-45591-4 ↗
53. Rolim, J., Mueller, F., Zomaya, A.Y., Ercal, F., Olariu, S., Ravindran, B., Gustafsson, J., Takada, H., Olsson, R., and Kale, L.V. (1999). An efficient and transparent thread migration scheme in the PM2 runtime system. Parallel and Distributed Processing, Springer.10.1007/BFb0097882 ↗
54. Hübner, M., and Becker, J. (2011). Invasive Computing: An Overview. Multiprocessor System-on-Chip: Hardware Design and Tool Integration, Springer.10.1007/978-1-4419-6460-1 ↗
55. Pujari, R.K., Wild, T., Herkersdorf, A., Vogel, B., and Henkel, J. (2011, January 12–14). Hardware assisted thread assignment for RISC based MPSoCs in invasive computing. Proceedings of the 2011 International Symposium on Integrated Circuits, Singapore.10.1109/ISICir.2011.6131920 ↗
56. Oechslein, B., Schedel, J., Kleinöder, J., Bauer, L., Henkel, J., Lohmann, D., and Schröder-Preikschat, W. (2011, January 10). OctoPOS: A Parallel Operating System for Invasive Computing. Proceedings of the EuroSys 2011 Workshop on Systems for Future Multi-Core Architectures (SFMA), Salzburg, Austria.
57. Saraswat, V., Bloom, B., Peshansky, I., Tardieu, O., and Grove, D. (2022, March 03). X10 Language Specification, Version 2.3. Available online: http://x10.sourceforge.net/documentation/languagespec/x10-230.pdf.
58. Calin, G., Derevenetc, E., Majumdar, R., and Meyer, R. (2013). A Theory of Partitioned Global Address Spaces. arXiv, arXiv:1307.6590.
59. Saraswat, V., Almasi, G., Bikshandi, G., Cascaval, C., Cunningham, D., Grove, D., Kodali, S., Peshansky, I., and Tardieu, O. (2010, January 6). The asynchronous partitioned global address space model. Proceedings of the First Workshop on Advances in Message Passing, Toronto, ON, Canada.
60. Bungartz, H.J., Riesinger, C., Schreiber, M., Snelting, G., and Zwinkau, A. (2013, January 16–19). Invasive computing in HPC with X10. Proceedings of the third ACM SIGPLAN X10 Workshop, Seattle, WA, USA.10.1145/2481268.2481274 ↗
61. Zwinkau, A., Buchwald, S., and Snelting, G. (2013). Invade X10 Documentation v0.5, Karlsruhe Institute of Technology. Technical Report.
62. Malleable invasive applications
63. Ribeiro, F.S., Nascimento, A.P., Boeres, C., Rebello, V.E.F., and Sena, A.C. (2013, January 23–26). Autonomic Malleability in Iterative MPI Applications. Proceedings of the Symposium on Computer Architecture and High Performance Computing, Washington, DC, USA.
64. EasyGrid: Towards a framework for the automatic Grid enabling of legacy MPI applications10.1002/cpe.821 ↗
65. Autonomic application management for large scale MPI programs10.1504/IJHPCN.2008.022299 ↗
66. Houzeaux, G., Badia, R.M., Borrell, R., Dosimont, D., Ejarque, J., Garcia-Gasulla, M., and López, V. (2021). Dynamic Resource Allocation for Efficient Parallel CFD Simulations, Barcelona Supercomputing Center. Technical Report.10.1016/j.compfluid.2022.105577 ↗
67. Alya: Multiphysics engineering simulation toward exascale10.1016/j.jocs.2015.12.007 ↗
68. COMP Superscalar, an interoperable programming framework10.1016/j.softx.2015.10.004 ↗
69. Tejedor, E., and Badia, R.M. (2008, January 19–22). COMP Superscalar: Bringing GRID superscalar and GCM together. Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid (CCGRID), Lyon, France.10.1109/CCGRID.2008.104 ↗
70. Lopez, V., Ramirez Miranda, G., and Garcia-Gasulla, M. (2021, January 25). TALP: A Lightweight Tool to Unveil Parallel Efficiency of Large-Scale Executions. Proceedings of the 2021 on Performance EngineeRing, Modelling, Analysis, and VisualizatiOn STrategy, Stockholm, Sweden.10.1145/3452412.3462753 ↗
71. ServiceSs: An Interoperable Programming Framework for the Cloud10.1007/s10723-013-9272-5 ↗
72. PyCOMPSs: Parallel computational workflows in Python10.1177/1094342015594678 ↗
73. Evaluating Data Redistribution in PaRSEC10.1109/TPDS.2021.3131657 ↗
74. Hints to Improve Automatic Load Balancing with LeWI for Hybrid Applications10.1016/j.jpdc.2014.05.004 ↗
75. Martín-Álvarez, I., Aliaga, J.I., Castillo, M.I., Mayo, R., and Iserte, S. (2021, January 7–10). Malleability Implementation in a MPI Iterative Method. Proceedings of the 2021 IEEE International Conference on Cluster Computing (CLUSTER), Portland, OR, USA.10.1109/Cluster48925.2021.00078 ↗
76. Zheng, G., Xiang, N., and Kale, L.V. (2012, January 25–28). A scalable double in-memory checkpoint and restart scheme towards exascale. Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN 2012), Boston, MA, USA.10.1109/DSNW.2012.6264677 ↗
77. Garcia, M., Corbalan, J., and Labarta, J. (2009, January 22–25). LeWI: A Runtime Balancing Algorithm for Nested Parallelism. Proceedings of the 2009 International Conference on Parallel Processing, Piscataway, NJ, USA.10.1109/ICPP.2009.56 ↗