Pipelined Dynamic Scheduling of Big Data Streams
Source abstract
Abstract
We are currently living in the big data era, in which it has become more necessary than ever to develop “smart” schedulers. It is common knowledge that the default Storm scheduler, as well as a large number of static schemes, has presented certain deficiencies. One of the most important of these deficiencies is the weakness in handling cases in which system changes occur. In such a scenario, some type of re-scheduling is necessary to keep the system working in the most efficient way. In this paper, we present a pipeline-based dynamic modular arithmetic-based scheduler (PMOD scheduler), which can be used to re-schedule the streams distributed among a set of nodes and their tasks, when the system parameters (number of tasks, executors or nodes) change. The PMOD scheduler organizes all the required operations in a pipeline scheme, thus reducing the overall processing time.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Cloud Computing and Resource Management · Distributed and Parallel Computing Systems · Advanced Database Systems and Queries
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Applied Informatics, School of Information Sciences, University of Macedonia Thessaloniki, 54616 Thessaloniki, Greece
Department of Early Childhood Education, Faculty of Education, University of Western Macedonia, 21, 53100 Florina, Greece
Funding
University of Western Macedonia
References (26)
Show source-provided references
1. Tantalaki, N., Souravlas, S., and Roumeliotis, M. (2019). A review on Big Data real-time stream processing and its scheduling techniques. Int. J. Parallel Emerg. Distrib. Syst.10.1080/17445760.2019.1585848 ↗
2. Tantalaki, N., Souravlas, S., Roumeliotis, M., and Katsavounis, S. (2019). Linear scheduling of big data streams on multiprocessor sets in the cloud. Ser. WI’19, Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, Thessaloniki, Greece, 14–17 October 2019, ACM.10.1145/3350546.3352507 ↗
3. Apache Software Foundation (2019, June 05). Apache Storm. Available online: http://storm.apache.org/.
4. Apache Software Foundation (2019, June 05). Spark Streaming-Apache Spark. Available online: http://spark.apache.org/streaming/.
5. Apache Software Foundation (2019, June 05). Apache Samza—A Distributed Stream Processing Framework. Available online: https://samza.apache.org.
6. Apache Software Foundation (2019, June 05). Apache Flink-Stateful Computations over Data Streams. Available online: https://flink.apache.org/.
7. Souravlas, S. (2019). ProMo: A Probabilistic Model for Dynamic Load-Balanced Scheduling of Data Flows in Cloud Systems. Electronics, 8.10.3390/electronics8090990 ↗
8. Aniello, L., Baldoni, R., and Querzoni, L. (July, January 29). Adaptive Online Scheduling in Storm. Proceedings of the 7th ACM International Conference on Distributed Event-based Systems (DEBS ’13), Arlington, TX, USA.
9. Eskandari, L., Huang, Z., and Eyers, D. (2016, January 2–5). P-Scheduler: AdaptiveHierarchical Scheduling in Apache Storm. Proceedings of the Australasian Computer Science Week Multiconference (ACSW ’16), Canberra, Australia. Article 26.10.1145/2843043.2843056 ↗
10. Eskandari, L., Mair, J., Huang, Z., and Eyers, D. (2018). Iterative scheduling for distributed stream processing systems. Ser.DEBS ’18, Proceedings of the 12th ACM International Conference on Distributed and Event-Based Systems, Hamilton, New Zealand, 25–29 June 2018, ACM.10.1145/3210284.3219768 ↗
11. Model-driven scheduling for dis-tributed stream processing systems10.1016/j.jpdc.2018.02.003 ↗
12. Eidenbenz, R., and Locher, T. (2016, January 10–14). Task allocation for distributed stream pro-cessing. Proceedings of the IEEE INFOCOM 2016—The 35th Annual IEEE InternationalConference on Computer Communications, San Francisco, CA, USA.10.1109/INFOCOM.2016.7524433 ↗
13. Xiang, D., Wu, Y., Shang, P., Jiang, J., Wu, J., and Yu, K. (2017, January 9–13). Rb-storm: Resourcebalance scheduling in Apache storm. Proceedings of the 2017 6th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), Hamamatsu, Japan.10.1109/IIAI-AAI.2017.63 ↗
14. Al-Sinayyid, A., and Zhu, M. (2020). Job scheduler for streaming applications inheterogeneous distributed processing systems. J. Supercomput.10.1007/s11227-020-03223-z ↗
15. Janssen, G., Verbitskiy, I., Renner, T., and Thamsen, L. (2018, January 10–13). Scheduling streamprocessing tasks on geo-distributed heterogeneous resources. Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.10.1109/BigData.2018.8622651 ↗
16. Peng, B., Hosseini, M., Hong, Z., Farivar, R., and Campbell, R. (2015, January 7–11). R-storm: Resource-aware scheduling in storm. Proceedings of the 16th Annual Middleware Conference, Ser. Middleware ’15, Vancouver, BC, Canada.10.1145/2814576.2814808 ↗
17. Performance-aware schedulingof streaming applications using genetic algorithm10.1016/j.procs.2017.05.249 ↗
18. Fu, T.Z.J., Ding, J., Ma, R.T.B., Winslett, M., Yang, Y., and Zhang, Z. (July, January 29). DRS: Dynamic Resource Scheduling for Real-Time Analyticsover Fast Streams. Proceedings of the IEEE 35th International Conference on Distributed Computing Systems, Columbus, OH, USA.
19. Optimal Operator Replication and Placement for Distributed Stream Processing Systems10.1145/3092819.3092823 ↗
20. Xu, J., Chen, Z., Tang, J., and Su, S. (July, January 30). T-Storm: Traffic-AwareOnline Scheduling in Storm. Proceedings of the 2014 IEEE 34th International Conference on Distributed Computing Systems (ICDCS ’14), Madrid, Spain.10.1109/ICDCS.2014.61 ↗
21. Chen, M.-M., Zhuang, C., Li, Z., and Xu, K.-F. (2014, January 12–14). A Task Scheduling Approach for Real-Time Stream Processing. Proceedings of the International Conference on Cloud Computing and Big Data, Wuhan, China.
22. A dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services10.1007/s00779-016-0914-z ↗
23. Dhalion: Self-regulating Stream Processing10.14778/3137765.3137786 ↗
24. A pipeline technique for dynamic data transfer on a multiprocessor grid10.1023/B:IJPP.0000038068.80639.52 ↗
25. On further reducing the cost of parallel pipelined message broadcasts10.1080/00207160600845203 ↗
26. Scheduling array redistribution with virtual channel support10.1007/s11227-015-1519-4 ↗