More on Pipelined Dynamic Scheduling of Big Data Streams
Source abstract
Abstract
An important as well as challenging task in modern applications is the management and processing with very short delays of large data volumes. It is quite often, that the capabilities of individual machines are exceeded when trying to manage such large data volumes. In this regard, it is important to develop efficient task scheduling algorithms, which reduce the stream processing costs. What makes the situation more difficult is the fact that the applications as well as the processing systems are prone to changes during runtime: processing nodes may be down, temporarily or permanently, more resources may be needed by an application, and so on. Therefore, it is necessary to develop dynamic schedulers, which can effectively deal with these changes during runtime. In this work, we provide a fast and fair task migration policy while maintaining load balancing and low latency times. The experimental results have shown that our scheme offers better load balancing and reduces the overall latency compared to the state of the art strategies, due to the stepwise communication and the pipeline based processing it employs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Cloud Computing and Resource Management · Distributed and Parallel Computing Systems · Advanced Data Storage Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Applied Informatics, University of Macedonia, 54616 Thessaloniki, Greece
School of Information Sciences, University of Macedonia, 54616 Thessaloniki, Greece
Department of Early Childhood Education, Faculty of Education, University of Western Macedonia, 21, 53100 Florina, Greece
Department of Production and Management Engineering, Democritus University of Thrace, 67150 Xanthi, Greece
Funding
University of Western Macedonia
References (31)
Show source-provided references
1. Souravlas, S., and Anastasiadou, S. (2020). Pipelined Dynamic Scheduling of Big Data Streams. Appl. Sci., 10.10.3390/app10144796 ↗
2. Tantalaki, N., Souravlas, S., and Roumeliotis, M. (2019). A review on Big Data real-time stream processing and its scheduling techniques. Int. J. Parallel Emerg. Distrib. Syst.10.1080/17445760.2019.1585848 ↗
3. Tantalaki, N., Souravlas, S., Roumeliotis, M., and Katsavounis, S. (2019, January 14–17). Linear scheduling of big data streams on multiprocessor sets in the cloud. Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, Thessaloniki, Greece.10.1145/3350546.3352507 ↗
4. Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud10.1109/ACCESS.2020.3004612 ↗
5. Wu, Y., and Tan, K.L. (2015, January 13–16). ChronoStream: Elastic stateful stream computation in the cloud. Proceedings of the 2015 IEEE 31st International Conference on Data Engineering, Seoul, Korea.10.1109/ICDE.2015.7113328 ↗
6. Janssen, G., Verbitskiy, I., Renner, T., and Thamsen, L. (2018, January 2–7). Scheduling streamprocessing tasks on geo-distributed heterogeneous resources. Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Los Alamitos, CA, USA.10.1109/BigData.2018.8622651 ↗
7. Eskandari, L., Mair, J., Huang, Z., and Eyers, D. (2020, January 13–17). Iterative scheduling for distributed stream processing systems. Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems, Montreal, QC, Canada.
8. Chen, M.-M., Zhuang, C., Li, Z., and Xu, K.-F. (2014, January 12–14). A Task Scheduling Approach for Real-Time Stream Processin. Proceedings of the International Conference on Cloud Computing and Big Data, Sydney, Australia.
9. Pahl, C., and Lee, B. (2015, January 24–26). Containers and clusters for edge cloud architectures—A technology review. Proceedings of the 3rd International Conference on Future Internet of Things and Cloud, Rome, Italy.10.1109/FiCloud.2015.35 ↗
10. Change detection in streaming data in the era of big data: Models and issues10.1145/2674026.2674031 ↗
11. Rethinking elastic online scheduling of big data streaming applications over high-velocity continuous data streams10.1007/s11227-017-2151-2 ↗
12. Pipelined fission for stream programs with dynamic selectivity and partitioned state10.1016/j.jpdc.2016.05.003 ↗
13. Sattler, K.-U., and Beier, F. (2013, January 30). Towards elastic stream processing: Patterns and infrastructure. Proceedings of the 1st International Workshop on Big Dynamic Distributed Data (BD3), Trento, Italy.
14. Fernandez, R.C., Migliavacca, M., Kalyvianaki, E., and Pietzuch, P. (2013, January 22–27). Integrating scale out and fault tolerance in stream processing using operator state management. Proceedings of the ACM SIGMOD International Conference on Management of Data, New York, NY, USA.
15. Heinze, T., Jerzak, Z., Hackenbroich, G., and Fetzer, C. (2014, January 26–29). Latency-aware elastic scaling for distributed data stream processing systems. Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, Mumbai, India.10.1145/2611286.2611294 ↗
16. Lohrmann, B., Janacik, P., and Kao, O. (2, January June). Elastic stream processing with latency guarantees. Proceedings of the 35th IEEE International Conference on Distributed Computing Systems (ICDCS), Columbus, OH, USA.
17. StreamCloud: An elastic and scalable data streaming system10.1109/TPDS.2012.24 ↗
18. Maeis, T.D., and Mencagli, G. (2016, January 12–16). Keep Calm and React with Foresight: Strategies for Low-latency and Energy-e†cient Elastic Data Stream Processing. Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’16), Barcelona, Spain.
19. (2017). Tiziano De Maeis and Gabriele Mencagli, Proactive elasticity and energy awareness in data stream processing. J. Syst. Softw., 127, 302–319.10.1016/j.jss.2016.08.037 ↗
20. Xu, J., Chen, Z., Tang, J., and Su, S. (July, January 30). T-Storm: Traffic-aware online scheduling in storm. Proceedings of the IEEE 34th International Conference on Distributed Computing Systems (ICDCS), Madrid, Spain.
21. Xu, L., Peng, B., and Gupta, I. (2016, January 4–8). Stela: Enabling stream processing systems to scale-in and scale-out on-demand. Proceedings of the IEEE International Conference on Cloud Engineering (IC2E 2016), Berlin, Germany.10.1109/IC2E.2016.38 ↗
22. Shukla, A., and Simmhan, Y. (2018, January 2–6). Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds. Proceedings of the 2018 IEEE 38th International Conference on Distributed Computing Systems, Vienna, Austria.10.1109/ICDCS.2018.00109 ↗
23. Ding, J., Fu, T.Z.J., Ma, R.T.B., Winslett, M., Yang, Y., Zhang, Z., and Chao, H. (2015). Optimal Operator State Migration for Elastic Data Stream Processing. Comput. Repos. (CoRR), abs/1501.03619.
24. Zhang, Z., Jin, P., Wang, X., Liu, R., and Wan, S. (2019, January 10–12). N-Storm: Efficient Thread-Level Task Migration in Apache Storm. Proceedings of the IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Zhangjiajie, China.10.1109/HPCC/SmartCity/DSS.2019.00219 ↗
25. Optimal operator deployment and replication for elastic distributed data stream processing10.1002/cpe.4334 ↗
26. Ma, K., Yang, B., and Yu, Z. (2017). Optimization of stream-based live data migration strategy in the cloud. Concurr. Comput. Pract. Exp.10.1002/cpe.4293 ↗
27. Liu, X., Harwood, A., Karunasekera, S., Rubinstein, B., and Buyya, R. (2017, January 14–17). E-storm: Replication-based state management in distributed stream processing systems. Proceedings of the 46th IEEE International conference on Parallel Processing (ICPP), Bristol, UK.10.1109/ICPP.2017.66 ↗
28. Saurez, E., Hong, K., Lillethun, D., Ramachandran, U., and Ottenwälder, B. (2016, January 20–24). Incremental deployment and migration of geo-distributed situation awareness applications in the fog. Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems, Irvine, CA, USA.10.1145/2933267.2933317 ↗
29. A pipeline technique for dynamic data transfer on a multiprocessor grid10.1023/B:IJPP.0000038068.80639.52 ↗
30. A Proof for the Queuing Formula L = λW10.1287/opre.9.3.383 ↗
31. Job scheduler for streaming applications inheterogeneous distributed processing systems10.1007/s11227-020-03223-z ↗