Cloud–Edge Hybrid Computing Architecture for Large-Scale Scientific Facilities Augmented with an Intelligent Scheduling System
Source abstract
Abstract
Synchrotron radiation sources are widely used in interdisciplinary research, generating an enormous amount of data while posing serious challenges to the storage, processing, and analysis capabilities of the large-scale scientific facilities worldwide. A flexible and scalable computing architecture, suitable for complex application scenarios, combined with efficient and intelligent scheduling strategies, plays a key role in addressing these issues. In this work, we present a novel cloud–edge hybrid intelligent system (CEHIS), which was architected, developed, and deployed by the Big Data Science Center (BDSC) at the Shanghai Synchrotron Radiation Facility (SSRF) and meets the computational needs of the large-scale scientific facilities. Our methodical simulations demonstrate that the CEHIS is more efficient and performs better than the cloud-based model. Here, we have applied a deep reinforcement learning approach to the task scheduling system, finding that it effectively reduces the total time required for the task completion. Our findings prove that the cloud–edge hybrid intelligent architectures are a viable solution to address the requirements and conditions of the modern synchrotron radiation facilities, further enhancing their data processing and analysis capabilities.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Cloud Computing and Resource Management · Distributed and Parallel Computing Systems · Scientific Computing and Data Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Shanghai Synchrotron Radiation Facility, Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai 201210, China
Shanghai Institute of Applied Physics, Chinese Academy of Sciences, Shanghai 201204, China
School of Physical Science and Technology, ShanghaiTech University, Shanghai 201210, China
Funding
Photon Science Research Center for Carbon Dioxide, CAS · Youth Innovation Promotion Association · Natural Science Foundation of Shanghai
References (33)
Show source-provided references
1. Synchrotron Big Data Science10.1002/smll.201802291 ↗
2. Beyond the Data Deluge10.1126/science.1170411 ↗
3. Pralavorio, C. (2015). LHC Season 2: CERN Computing Ready for Data Torrent, CERN.
4. (2019, May 01). FLIR Systems. Available online: https://www.flir.com/products/oryx-10gige.
5. Outlook for artificial intelligence and machine learning at the NSLS-II10.1088/2632-2153/abbd4e ↗
6. Advancing Discovery with Artificial Intelligence and Machine Learning at NSLS-II10.1080/08940886.2022.2114716 ↗
7. The design of a data management system at HEPS10.1107/S1600577520015167 ↗
8. Real-time data-intensive computing10.1063/1.4952921 ↗
9. Bard, D., Snavely, C., Gerhardt, L.M., Lee, J., Totzke, B., Antypas, K., Arndt, W., Blaschke, J.P., Byna, S., and Cheema, R. (2022). The LBNL Superfacility Project Report. arXiv.10.2172/1875256 ↗
10. Bashor, J. (1999). NERSC and ESnet: 25 Years of Leadership.
11. Blaschke, J., Brewster, A.S., Paley, D.W., Mendez, D., Sauter, N.K., Kröger, W., Shankar, M., Enders, B., and Bard, D.J. (2021). Real-Time XFEL Data Analysis at SLAC and NERSC: A Trial Run of Nascent Exascale Experimental Data Analysis. arXiv.
12. Giannakou, A., Blaschke, J.P., Bard, D., and Ramakrishnan, L. (2021, January 19). Experiences with Cross-Facility Real-Time Light Source Data Analysis Workflows. Proceedings of the 2021 IEEE/ACM HPC for Urgent Decision Making (UrgentHPC), St. Louis, MO, USA.10.1109/UrgentHPC54802.2021.00011 ↗
13. Vescovi, R., Chard, R., Saint, N., Blaiszik, B., Pruyne, J., Bicer, T., Lavens, A., Liu, Z., Papka, M.E., and Narayanan, S. (2022). Linking Scientific Instruments and HPC: Patterns, Technologies, Experiences. arXiv.10.2139/ssrn.4141629 ↗
14. Enders, B., Bard, D., Snavely, C., Gerhardt, L.M., Lee, J.R., Totzke, B., Antypas, K., Byna, S., Cheema, R., and Cholia, S. (2020, January 12). Cross-facility Science with the Superfacility Project at LBNL. Proceedings of the 2020 IEEE/ACM 2nd Annual Workshop on Extreme-Scale Experiment-in-the-Loop Computing (XLOOP), Atlanta, GA, USA.10.1109/XLOOP51963.2020.00006 ↗
15. Deslippe, J.R., Essiari, A., Patton, S.J., Samak, T., Tull, C.E., Hexemer, A., Kumar, D., Parkinson, D.Y., and Stewart, P. (2014, January 16). Workflow Management for Real-Time Analysis of Lightsource Experiments. Proceedings of the 2014 9th Workshop on Workflows in Support of Large-Scale Science, New Orleans, LA, USA.10.1109/WORKS.2014.9 ↗
16. GigaFRoST: The gigabit fast readout system for tomography10.1107/S1600577517013522 ↗
17. Real-time reconstruction and visualisation towards dynamic feedback control during time-resolved tomography experiments at TOMCAT10.1038/s41598-019-54647-4 ↗
18. Towards on-the-fly data post-processing for real-time tomographic imaging at TOMCAT10.1186/s40679-016-0035-9 ↗
19. TomoPy: A framework for the analysis of synchrotron tomographic data10.1107/S1600577514013939 ↗
20. Xi-cam: A versatile interface for data visualization and analysis10.1107/S1600577518005787 ↗
21. Aquarium: An automatic data-processing and experiment information management system for biological macromolecular crystallography beamlines10.1107/S1600576719001183 ↗
22. Shanghai Synchrotron Radiation Facility10.1007/s11434-009-0689-y ↗
23. Shanghai synchrotron radiation facility10.1093/nsr/nwt039 ↗
24. Progress and Future of Shanghai Synchrotron Radiation Facility10.3131/jvsj2.59.198 ↗
25. Deploying the Big Data Science Center at the Shanghai Synchrotron Radiation Facility: The first superfacility platform in China10.1088/2632-2153/abe193 ↗
26. Sun, B., Wang, Y., Liu, K., Wang, Q., and He, J. (2018, January 11–15). Design of new sub-micron protein crystallography beamline at SSRF. Proceedings of the 13th International Conference on Synchrotron Radiation Instrumentation, Taipei, Taiwan.10.1063/1.5084659 ↗
27. Li, Z., Fan, Y., Xue, L., Zhang, Z., and Wang, J. (2018, January 11–15). The design of the test beamline at SSRF. Proceedings of the 13th International Conference on Synchrotron Radiation Instrumentation, Taipei, Taiwan.10.1063/1.5084671 ↗
28. Edge Computing: Vision and Challenges10.1109/JIOT.2016.2579198 ↗
29. Heterogeneous edge computing open platforms and tools for internet of things10.1016/j.future.2019.12.036 ↗
30. Yin, J., Zhang, G., Cao, H., Dash, S., Chakoumakos, B.C., and Wang, F. (2022, January 23–25). Toward an Autonomous Workflow for Single Crystal Neutron Diffraction. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Virtual Event.10.1007/978-3-031-23606-8_15 ↗
31. Hirschman, J., Kamalov, A., Obaid, R., O’Shea, F.H., and Coffee, R.N. (2022, January 23–25). At-the-Edge Data Processing for Low Latency High Throughput Machine Learning Algorithms. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Virtual Event.10.1007/978-3-031-23606-8_7 ↗
32. Reinforcement Learning: An Introduction10.1109/TNN.2004.842673 ↗
33. Sinaei, K., and Yazdi, M.R.S. (2021, January 25–27). PID Controller Tuning with Deep Reinforcement Learning Policy Gradient Methods. Proceedings of the 29th Intermational Conference of Iranian Society of Mechanical Engineers & 8th Conference on Thermal Power Plants, Tehran, Iran.