Thread-Aware Mechanism to Enhance Inter-Node Load Balancing for Multithreaded Applications on NUMA Systems
Source abstract
Abstract
NUMA multi-core systems divide system resources into several nodes. When an imbalance in the load between cores occurs, the kernel scheduler’s load balancing mechanism then migrates threads between cores or across NUMA nodes. Remote memory access is required for a thread to access memory on the previous node, which degrades performance. Threads to be migrated must be selected effectively and efficiently since the related operations run in the critical path of the kernel scheduler. This study focuses on improving inter-node load balancing for multithreaded applications. We propose a thread-aware selection policy that considers the distribution of threads on nodes for each thread group while migrating one thread for inter-node load balancing. The thread is selected for which its thread group has the least exclusive thread distribution, and thread members are distributed more evenly on nodes. This has less influence on data mapping and thread mapping for the thread group. We further devise several enhancements to eliminate superfluous evaluations for multithreaded processes, so the selection procedure is more efficient. The experimental results for the commonly used PARSEC 3.0 benchmark suite show that the modified Linux kernel with the proposed selection policy increases performance by 10.7% compared with the unmodified Linux kernel.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Parallel Computing and Optimization Techniques · Distributed and Parallel Computing Systems · Cloud Computing and Resource Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Management, National Chi Nan University, Puli 54516, Taiwan
Funding
Ministry of Science and Technology, Taiwan
References (21)
Show source-provided references
1. Bovet, D.P., and Cesati, M. (2005). Understanding the Linux Kernel, O’Reilly Media Inc.. [3rd ed.].
2. An Overview of Non-Uniform Memory Access10.1145/2500468.2500477 ↗
3. Chiang, M.L., Tu, S.W., Su, W.L., and Lin, C.W. (2018, January 23–27). Enhancing Inter-Node Process Migration for Load Balancing on Linux-based NUMA Multicore Systems. Proceedings of the 10th IEEE International Workshop on Computer Forensics in Software Engineering, Tokyo, Japan.10.1109/COMPSAC.2018.10264 ↗
4. Memory-Aware Kernel Mechanism and Policies for Improving Inter-Node Load Balancing on NUMA Systems
5. Chen, J., Banerjee, S.S., Kalbarczyk, Z.T., and Iyer, R.K. (2020, January 24–25). Machine Learning for Load Balancing in the Linux Kernel. Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop on Systems, Tsukuba, Japan.10.1145/3409963.3410492 ↗
6. (2021, June 01). PARSEC Benchmark Suite. Available online: http://parsec.cs.princeton.edu/.
7. Kernel-Based Thread and Data Mapping for Improved Memory Affinity10.1109/TPDS.2015.2504985 ↗
8. Gennaro, I.D., Pellegrini, A., and Quaglia, F. (2016, January 16–19). OS-based NUMA Optimization: Tackling the Case of Truly Multithread Applications with Non-partitioned Virtual Page Accesses. Proceedings of the IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, Cartagena, Colombia.
9. Lepers, B., Quéma, V., and Fedorova, A. (2015, January 8–10). Thread and Memory Placement on NUMA Systems: Asymmetry Matters. Proceedings of the 2015 USENIX Annual Technical Conference, Santa Clara, CA, USA.
10. Li, T., Baumberger, D., Koufaty, D.A., and Hahn, S. (2007, January 10–16). Efficient Operating System Scheduling for Performance-asymmetric Multi-core Architectures. Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, Reno, NV, USA.10.1145/1362622.1362694 ↗
11. Terboven, C., Mey, D.A.A., Schmidl, D., Jin, H., and Reichstein, T. (2008, January 5). Data and Thread Affinity in OpenMP Programs. Proceedings of the 2008 Workshop on Memory Access on Future Processors, Ischia, Italy.10.1145/1366219.1366222 ↗
12. Trends in Data Locality Abstractions for HPC Systems10.1109/TPDS.2017.2703149 ↗
13. Goglin, B., and Furmento, N. (2009, January 13–17). Memory Migration on Next-touch. Proceedings of the Linux Symposium, Montreal, QC, Canada.
14. Goglin, B., and Furmento, N. (2009, January 23–29). Enabling High-Performance Memory Migration for Multithreaded Applications on Linux. Proceedings of the IEEE International Symposium on Parallel & Distributed Processing, Rome, Italy.10.1109/IPDPS.2009.5161101 ↗
15. Mishra, V.K., and Mehta, D.A. (2013, January 22–23). Performance Enhancement of NUMA Multiprocessor Systems with On-Demand Memory Migration. Proceedings of the 2013 IEEE 3rd International Advance Computing Conference, Ghaziabad, India.10.1109/IAdCC.2013.6506812 ↗
16. Marchetti, M., Kontothanassis, L., Bianchini, R., and Scott, M. (1995, January 25–28). Using Simple Page Placement Policies to Reduce the Cost of Cache Fills in Coherent Shared-memory Systems. Proceedings of the 9th International Parallel Processing Symposium, Santa Barbara, CA, USA.10.21236/ADA289887 ↗
17. Schermerhorn, L.T. (, January 15–20). Automatic Page Migration for Linux. Proceedings of the Linux Symposium, Sydney, Australia.
18. (2021, June 01). Supermicro AS 4042G-72RF4. Available online: https://www.supermicro.com/Aplus/system/Tower/4042/AS-4042G-72RF4.cfm.
19. (2021, June 01). AMD Opteron 6320 Processor. Available online: https://www.amd.com/en/products/cpu/6320.
20. (2021, June 01). The Linux Kernel Archives. Available online: https://www.kernel.org/pub/linux/kernel/v3.0/.
21. (2021, July 04). Linux CPUFreq Governors—Information for Users and Developers. Available online: https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt.