A Survey of Spark Scheduling Strategy Optimization Techniques and Development Trends
Source abstract
Abstract
Spark performs excellently in large-scale data-parallel computing and iterative processing. However, with the increase in data size and program complexity, the default scheduling strategy has difficulty meeting the demands of... | Find, read and cite all the research you need on Tech Science Press
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Scheduling and Optimization Algorithms · Cloud Computing and Resource Management · IoT and Edge/Fog Computing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (75)
Show source-provided references
1. Comparative study on MapReduce and spark for big data analytics
2. Apache spark: a unified engine for big data processing10.1145/2934664 ↗
3. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
4. Spark: cluster computing with working sets
5. Leveraging resource management for efficient performance of Apache Spark10.1186/s40537-019-0240-1 ↗
6. A survey on spark ecosystem: big data processing infrastructure, machine learning, and applications
7. Survey on performance optimization technologies for spark
8. Survey on storage and optimization techniques of HDFS
9. Survey on JVM optimization for big data processing frameworks
10. Zaharia · An architecture for fast and general data processing on large clusters · 201610.1145/2886107 ↗
11. Parallel machine learning algorithm using fine-grained-mode spark on a mesos big data cloud computing software framework for mobile robotic intelligent fault recognition10.1109/ACCESS.2020.3007499 ↗
12. DAG-aware joint task scheduling and cache management in spark clusters
13. A hard real-time scheduler for spark on YARN
14. Efficient real-time earliest deadline first based scheduling for Apache spark
15. Exploring the fairness and resource distribution in an Apache mesos environment
16. Multicore architecture speedup computation based on Amdahl’s law and Rent’s rule
17. PAS: performance-aware job scheduling for big data processing systems
18. Parallelism deduction algorithm for spark
19. Spatiotemporal data partitioning for distributed random forest algorithm: air quality prediction using imbalanced big spatiotemporal data on spark distributed framework10.1016/j.eti.2022.102776 ↗
20. An optimal locality-aware task scheduling algorithm based on bipartite graph modelling for spark applications10.1109/TPDS.2020.2992073 ↗
21. Adaptive scheduling strategy based on deadline under cloud platform
22. Design of distributed computing framework for foreign exchange market monitoring
23. Scheduling-guided automatic processing of massive hyperspectral image classification on cloud computing architectures10.1109/TCYB.2020.3026673 ↗
24. Multiobjective task scheduling for energy-efficient cloud implementation of hyperspectral image classification10.1109/JSTARS.2020.3036896 ↗
25. Semi-clairvoyant scheduling in data analytics systems10.1109/TC.2019.2906193 ↗
26. Fine-grained dynamic resource allocation for big-data applications10.1109/TSE.2019.2931537 ↗
27. Performance improvement of DAG-aware task scheduling algorithms with efficient cache management in spark10.3390/electronics10161874 ↗
28. Scheduling spark tasks with data skew and deadline constraints10.1109/ACCESS.2020.3040719 ↗
29. PAC: a monitoring framework for performance analysis of compression algorithms in Spark10.1016/j.future.2024.02.009 ↗
30. DAG-aware harmonizing job scheduling and data caching for disaggregated analytics frameworks10.1016/j.future.2024.03.005 ↗
31. Task scheduling optimization in Spark environment with unbalanced resources
32. Dynamic resource provisioning for iterative workloads on Apache spark10.1109/TCC.2021.3108043 ↗
33. Dynamic data partition based on node load
34. Node priority optimization in distributed heterogeneous clusters
35. On-the-fly fusion of remotely-sensed big data using an elastic computing paradigm with a containerized spark engine on Kubernetes10.3390/s21092971 ↗
36. A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBench10.1186/s40537-020-00388-5 ↗
37. Heterogeneous cluster resource allocation algorithm considering application and node characteristics
38. A holistic cross-layer optimization approach for mitigating stragglers in in-memory data processing10.1016/j.sysarc.2020.101801 ↗
39. A data skew correction scheduling strategy of heterogeneous Spark cluster
40. Cost-efficient dynamic scheduling of big data applications in Apache spark on cloud10.1016/j.jss.2019.110515 ↗
41. Distributed parallel task scheduling on spark-gpu framework for oceanographic geospatial data processing
42. Adaptive priority-based data placement and multi-task scheduling in geo-distributed cloud systems10.1016/j.knosys.2021.107050 ↗
43. SimCost: cost-effective resource provision prediction and recommendation for spark workloads10.1007/s10619-023-07436-y ↗
44. Improving data locality of tasks by executor allocation in spark computing environment10.1109/TCC.2024.3406041 ↗
45. Distributed tree-based machine learning for short-term load forecasting with Apache spark10.1109/ACCESS.2021.3072609 ↗
46. Du · Comput Electr Eng · 2021 · 92 · 10709210.1016/j.compeleceng.2021.107092 ↗
47. Dynamic memory-aware scheduling in spark computing environment10.1016/j.jpdc.2020.03.010 ↗
48. Mary, Hugo, and Hugo*: learning to schedule distributed data-parallel processing jobs on shared clusters10.1002/cpe.5823 ↗
49. Workflows scheduling powered by execution time prediction model
50. Adaptive scheduling framework of streaming applications based on resource demand prediction with hybrid algorithms10.1007/s10723-024-09756-4 ↗
51. Hawkeye: adaptive straggler identification on heterogeneous spark cluster with reinforcement learning10.1109/ACCESS.2020.2982320 ↗
52. Distributed intelligent optimization of e-commerce user purchase data mining using spark framework10.31449/inf.v48i20.6779 ↗
53. Big data analytics for identifying electricity theft using machine learning approaches in microgrids for smart communities10.1002/cpe.6316 ↗
54. Mining area skyline objects from map-based big data using Apache Spark framework10.1016/j.array.2024.100373 ↗
55. A new Apache Spark-based framework for big data streaming forecasting in IoT networks10.1007/s11227-023-05100-x ↗
56. Apache spark in healthcare: advancing data-driven innovations and better patient care
57. An adaptive and real-time based architecture for financial data integration10.1186/s40537-019-0260-x ↗
58. A survey on software-hardware acceleration for fully homomorphic encryption
59. Electric load clustering in smart grid: methodologies, applications, and future trends10.35833/MPCE.2020.000472 ↗
60. Role of big data analytics in supply chain management: current trends and future perspectives.10.1080/00207543.2020.1793011 ↗
61. A parallelization model for performance characterization of Spark Big Data jobs on Hadoop clusters10.1186/s40537-021-00499-7 ↗
62. Runtime prediction of big data jobs: performance comparison of machine learning algorithms and analytical models10.1186/s40537-022-00623-1 ↗
63. Analyzing performance of Apache Tez and MapReduce with hadoop multinode cluster on Amazon cloud10.1186/s40537-016-0051-6 ↗
64. Edge computing: vision and challenges10.1109/JIOT.2016.2579198 ↗
65. Survey on the methods of computation offloading in mobile edge computing
66. Emerging high-performance computing systems and technology10.1360/N112016-00147 ↗
67. State-of-the-art survey of real-time multicore system
68. Scalability of blockchain: a comprehensive review and future research direction10.1007/s10586-023-04257-7 ↗
69. Byzantine fault-tolerant consensus algorithms: a survey10.3390/electronics12183801 ↗
70. Design and implementation of trusted fusion and sharing model of agricultural data based on Blockchain
71. Energy-aware task scheduling and offloading using deep reinforcement learning in SDN-enabled IoT network10.1016/j.comnet.2022.108957 ↗
72. Task scheduling and load balancing in SDN-based cloud computing: a review of relevant research
73. Salehnia · Handbook of whale optimization algorithm · 2024 · 10910.1016/B978-0-32-395365-8.00014-2 ↗
74. Borg: the next generation
75. Kubernetes scheduling: taxonomy, ongoing issues and challenges10.1145/3539606 ↗