Analyzing the Performance of the S3 Object Storage API for HPC Workloads
Source abstract
Abstract
The line between HPC and Cloud is getting blurry: Performance is still the main driver in HPC, while cloud storage systems are assumed to offer low latency, high throughput, high availability, and scalability. The Simple Storage Service S3 has emerged as the de facto storage API for object storage in the Cloud. This paper seeks to check if the S3 API is already a viable alternative for HPC access patterns in terms of performance or if further performance advancements are necessary. For this purpose: (a) We extend two common HPC I/O benchmarks—the IO500 and MD-Workbench—to quantify the performance of the S3 API. We perform the analysis on the Mistral supercomputer by launching the enhanced benchmarks against different S3 implementations: on-premises (Swift, MinIO) and in the Cloud (Google, IBM…). We find that these implementations do not yet meet the demanding performance and scalability expectations of HPC workloads. (b) We aim to identify the cause for the performance loss by systematically replacing parts of a popular S3 client library with lightweight replacements of lower stack components. The created S3Embedded library is highly scalable and leverages the shared cluster file systems of HPC infrastructure to accommodate arbitrary S3 client applications. Another introduced library, S3remote, uses TCP/IP for communication instead of HTTP; it provides a single local S3 gateway on each node. By broadening the scope of the IO500, this research enables the community to track the performance growth of S3 and encourage sharing best practices for performance optimization. The analysis also proves that there can be a performance convergence—at the storage level—between Cloud and HPC over time by using a high-performance S3 library like S3Embedded.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Data Storage Technologies · Cloud Computing and Resource Management · Distributed and Parallel Computing Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
MIN Faculty, University of Hamburg, 20146 Hamburg, Germany
Institute of Computer Science, Faculty of Mathematics and Computer Science, Georg-August-Universität Göttingen/GWDG, 37018 Göttingen, Germany
References (33)
Show source-provided references
1. AWS (2019, July 19). AWS S3. Available online: https://aws.amazon.com/de/s3/.
2. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D., and Maltzahn, C. (2006, January 6–8). Ceph: A scalable, high-performance distributed file system. Proceedings of the 7th Symposium on Operating Systems Design and Implementation, Seattle, WA, USA.
3. Foundation, O. (2020, September 19). OpenStack Swift. Available online: https://github.com/openstack/swift.
4. MinIO, I. (2020, September 19). Kubernetes Native, High Performance Object Storage. Available online: https://min.io.
5. NetCDF: An interface for scientific data access10.1109/38.56302 ↗
6. Lofstead, J.F., Klasky, S., Schwan, K., Podhorszki, N., and Jin, C. (2008, January 23). Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments, Boston, MA, USA.10.1145/1383529.1383533 ↗
7. Lofstead, J., Jimenez, I., Maltzahn, C., Koziol, Q., Bent, J., and Barton, E. (2016, January 13–18). DAOS and friends: A proposal for an exascale storage system. Proceedings of the SC’16 International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, USA.10.1109/SC.2016.49 ↗
8. Jamal, A., Fleiner, R., and Kail, E. (2021, January 19–21). Performance Comparison between S3, HDFS and RDS storage technologies for real-time big-data applications. Proceedings of the 2021 IEEE 15th International Symposium on Applied Computational Intelligence and Informatics (SACI), Timisoara, Romania.10.1109/SACI51354.2021.9465594 ↗
9. Milojicic, D., Faraboschi, P., Dube, N., and Roweth, D. (2021, January 1–5). Future of HPC: Diversifying Heterogeneity. Proceedings of the 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), Grenoble, France.10.23919/DATE51398.2021.9474063 ↗
10. (2020, December 09). S3EmbeddedLib. Available online: https://github.com/JulianKunkel/S3EmbeddedLib.
11. (2020, August 19). bji. libs3. Available online: https://github.com/bji/libs3.
12. Kunkel, J., Lofstead, G.F., and Bent, J. (2017). The Virtual Institute for I/O and the IO-500.
13. Kunkel, J.M., and Markomanolis, G.S. (2018, January 24–28). Understanding metadata latency with MDWorkbench. Proceedings of the International Conference on High Performance Computing, Frankfurt, Germany.10.1007/978-3-030-02465-9_5 ↗
14. DKRZ (2020, July 19). Mistral. Available online: https://www.dkrz.de/up/systems/mistral/configuration.
15. Garfinkel, S. (2007). An Evaluation of Amazon’s Grid Computing Services: EC2, S3, and SQS, Harvard Computer Science Group. Available online: http://nrs.harvard.edu/urn-3:HUL.InstRepos:24829568.
16. Palankar, M.R., Iamnitchi, A., Ripeanu, M., and Garfinkel, S. (2008, January 24). Amazon S3 for science grids: A viable solution?. Proceedings of the 2008 International Workshop on Data-Aware Distributed Computing, Boston, MA, USA.10.1145/1383519.1383526 ↗
17. DepSky: Dependable and secure storage in a cloud-of-clouds10.1145/2535929 ↗
18. Using S3 cloud storage with ROOT and CvmFS10.1088/1742-6596/664/2/022001 ↗
19. Understanding the performance and potential of cloud computing for scientific applications10.1109/TCC.2015.2404821 ↗
20. Google (2020, July 19). PerfKit Benchmarker. Available online: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.
21. Bjornson, Z. (2020, July 19). Cloud Storage Performance. Available online: https://blog.zachbjornson.com/2015/12/29/cloud-storage-performance.html.
22. Design and Evaluation of a Simple Data Interface for Efficient Data Transfer across Diverse Storage10.1145/3452007 ↗
23. Gadban, F., Kunkel, J., and Ludwig, T. (2020, January 22–25). Investigating the Overhead of the REST Protocol When Using Cloud Services for HPC Storage. Proceedings of the International Conference on High Performance Computing, Frankfurt am Main, Germany.10.1007/978-3-030-59851-8_10 ↗
24. Korolev, V. (2020, August 19). AWS4C—A C Lbrary to Interface with Amazon Web Services. Available online: https://github.com/vladistan/aws4c.
25. Welch, B., and Noer, G. (2013, January 6–10). Optimizing a hybrid SSD/HDD HPC storage system based on file size distributions. Proceedings of the 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), Long Beach, CA, USA.10.1109/MSST.2013.6558449 ↗
26. (2020, August 19). bji. libs3 Removes Support for Signature V2. Available online: https://github.com/bji/libs3/pull/50.
27. Braam, P. (2019). The Lustre storage architecture. arXiv.
28. Sysoev, I. (2020, July 19). Nginx. Available online: https://nginx.org.
29. LLNL (2020, September 19). IOR Parallel I/O Benchmarks. Available online: https://github.com/hpc/ior.
30. AWS (2020, July 19). Multipart Upload Overview. Available online: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html.
31. Walsdorf, O. (2020, September 19). Cisco UCS C240 M5 with Scality Ring. Available online: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_c240_m5_scalityring.html#_Toc15279751.
32. Zheng, Q., Chen, H., Wang, Y., Duan, J., and Huang, Z. (2012, January 24–29). Cosbench: A benchmark tool for cloud object storage services. Proceedings of the 2012 IEEE Fifth International Conference on Cloud Computing, Honolulu, HI, USA.10.1109/CLOUD.2012.52 ↗
33. AWS (2020, September 19). Performance Design Patterns for Amazon S3. Available online: https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance-design-patterns.html.