Algorithm and Implementation of Distributed ESN Using Spark Framework and Parallel PSO
Source abstract
Abstract
The echo state network (ESN) employs a huge reservoir with sparsely and randomly connected internal nodes and only trains the output weights, which avoids the suboptimal problem, exploding and vanishing gradients, high complexity and other disadvantages faced by traditional recurrent neural network (RNN) training. In light of the outstanding adaption to nonlinear dynamical systems, ESN has been applied into a wide range of applications. However, in the era of Big Data, with an enormous amount of data being generated continuously every day, the data are often distributed and stored in real applications, and thus the centralized ESN training process is prone to being technologically unsuitable. In order to achieve the requirement of Big Data applications in the real world, in this study we propose an algorithm and its implementation for distributed ESN training. The mentioned algorithm is based on the parallel particle swarm optimization (P-PSO) technique and the implementation uses Spark, a famous large-scale data processing framework. Four extremely large-scale datasets, including artificial benchmarks, real-world data and image data, are adopted to verify our framework on a stretchable platform. Experimental results indicate that the proposed work is accurate in the era of Big Data, regarding speed, accuracy and generalization capabilities.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Neural Networks and Reservoir Computing · Machine Learning and ELM · Neural Networks and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, China
Funding
the Fundamental Research Funds for Central Universities
References (56)
Show source-provided references
1. Scesn, spesn, swesn: Three recurrent neural echo state networks with clustered reservoirs for prediction of nonlinear and chaotic time series10.1007/s10489-015-0652-3 ↗
2. On the optimal structure design of multilayer feedforward neural networks for pattern recognition10.1142/S0218001402001812 ↗
3. Minimisation methods for training feedforward neural networks10.1016/0893-6080(94)90052-3 ↗
4. Wong, K.P. (1993, January 7–10). Artificial Intelligence and Neural Network Applications in Power Systems. Proceedings of the 2nd International Conference on Advances in Power System Control, Operation & Management, Hong Kong, China.
5. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
6. Echo state networks with filter neurons and a delay&sum readout
7. Review on echo state networks
8. Pascanu, R., Mikolov, T., and Bengio, Y. (2013, January 16–21). On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning (ICML 2013), Atlanta, GA, USA.
9. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note
10. Skowronski, M.D., and Harris, J.G. (2006, January 21–24). Minimum mean squared error time series classification using an echo state network prediction model. Proceedings of the ISCAS 2006: 2006 IEEE International Symposium on Circuits and Systems, Kos, Greece.
11. Modeling neural plasticity in echo state networks for classification and regression10.1016/j.ins.2015.11.017 ↗
12. Chaotic time series prediction based on a novel robust echo state network10.1109/TNNLS.2012.2188414 ↗
13. Prediction of the italian electricity price for smart grid applications10.1016/j.neucom.2015.02.089 ↗
14. Short-term electric load forecasting using echo state networks and pca decomposition10.1109/ACCESS.2015.2485943 ↗
15. Ding, H.Y., Pei, W.J., and He, Z.Y. (2005, January 28–30). A multiple objective optimization based echo state network tree and application to intrusion detection. Proceedings of the IEEE International Workshop on Vlsi Design and Video Technology, Suzhou, China.
16. Big data driven smart energy management: From big data to big insights10.1016/j.rser.2015.11.050 ↗
17. Big data challenge: A data management perspective10.1007/s11704-013-3903-7 ↗
18. A review of datasets and load forecasting techniques for smart natural gas and water grids: Analysis and experiments10.1016/j.neucom.2015.04.098 ↗
19. Factors important for good visualisation of time series
20. Handling big data: Research challenges and future directions10.1007/s11227-016-1677-z ↗
21. Distributed learning in wireless sensor networks10.1109/MSP.2006.1657817 ↗
22. Scardapane, S., Fierimonte, R., Wang, D., Panella, M., and Uncini, A. (2015, January 12–17). Distributed music classification using random vector functional-link nets. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2015), Killarney, Ireland.10.1109/IJCNN.2015.7280333 ↗
23. Parallel reservoir computing using optical amplifiers10.1109/TNN.2011.2161771 ↗
24. Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the IEEE International Conference on Neural Networks, Perth, Australia.
25. (2017, February 10). Apache Spark™. Available online: http://spark.apache.org/.
26. A parallel patient treatment time prediction algorithm and its applications in hospital queuing-recommendation in a big data environment10.1109/ACCESS.2016.2558199 ↗
27. Jaeger, H. (2002). Short Term Memory in Echo State Networks, Forschungszentrum Informationstechnik GmbH. GMD Report.
28. Cius, M.L.S., Dan, P., Jaeger, H., and Siewert, U. (2006). Time Warping Invariant Echo State Networks, International University Bremen.
29. Ridge regression learning in esn for chaotic time series prediction
30. Jaeger, H., and Jaeger, H. (2007). Discovering Multiscale Dynamical Features with Hierarchical Echo State Networks, Jacobs University Bremen.
31. A comparative study of four parallel and distributed pso methods10.1007/s00354-010-0102-z ↗
32. An optimal pso distributed precoding algorithm in qrd-based multi-relay system10.1016/j.future.2012.04.003 ↗
33. Map-reduce framework-based non-iterative granular echo state network for prediction intervals construction10.1016/j.neucom.2016.10.019 ↗
34. Distributed sparse linear regression10.1109/TSP.2010.2055862 ↗
35. Distributed optimization and statistical learning via the alternating direction method of multipliers
36. Convex optimization for big data: Scalable, randomized, and parallel algorithms for big data analytics10.1109/MSP.2014.2329397 ↗
37. A distributed algorithm for fitting generalized additive models10.1007/s11081-013-9215-9 ↗
38. Dean, J., Corrado, G.S., Monga, R., Chen, K., Devin, M., Le, Q.V., Mao, M.Z., Ranzato, A., Senior, A., and Tucker, P. (2012, January 3–8). Large scale distributed deep networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
39. A decentralized training algorithm for echo state networks in distributed big data applications10.1016/j.neunet.2015.07.006 ↗
40. Mapreduce: Simplified data processing on large clusters10.1145/1327452.1327492 ↗
41. Bigtable: A distributed storage system for structured data10.1145/1365815.1365816 ↗
42. Hall, K.B., Gilpin, S., and Mann, G. (2010, January 10–11). Mapreduce/bigtable for distributed optimization. Proceedings of the NIPS LCCC Workshop, Whistler, BC, Canada.
43. Parallel programming paradigms and frameworks in big data era10.1007/s10766-013-0272-7 ↗
44. Big data analytic architecture for intruder detection in heterogeneous wireless sensor networks10.1016/j.jnca.2016.03.004 ↗
45. Li, F., Li, G., Hwang, S.-W., Yao, B., and Zhang, Z. (2014). Time series classification using multi-channels deep convolutional neural networks. Web-Age Information Management, Proceedings of the 15th International Conference (WAIM 2014), Macau, China, 16–18 June 2014, Springer.
46. Lecun, Y., and Bengio, Y. (1998). Convolutional Networks for Images, Speech, and Time Series, MIT Press.
47. Cui, Z., Chen, W., and Chen, Y. (arXiv, 2016). Multi-scale convolutional neural networks for time series classification, arXiv.
48. Wang, Z., and Oates, T. (2015, January 25–30). Encoding time series as images for visual inspection and classification using tiled convolutional neural networks. Proceedings of the Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.
49. Dalto, M. (2015, January 17–19). Deep neural networks for time series prediction with applications in ultra-short-term wind forecasting. Proceedings of the 2015 IEEE International Conference on Industrial Technology, Seville, Spain.10.1109/ICIT.2015.7125335 ↗
50. Time series prediction using convolution sum discrete process neural network10.14311/NNW.2014.24.025 ↗
51. Aytar, Y., Vondrick, C., and Torralba, A. (2016, January 5–11). Soundnet: Learning sound representations from unlabeled video. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.10.1109/CVPR.2016.18 ↗
52. Oord, A.V.D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (arXiv, 2016). Wavenet: A generative model for raw audio, arXiv.
53. Reservoir computing approaches to recurrent neural network training10.1016/j.cosrev.2009.03.005 ↗
54. Jaeger, H. (2002, January 9–14). Adaptive nonlinear system identification with echo state networks. Proceedings of the 16th Annual Neural Information Processing Systems Conference (NIPS 2002), Vancouver, BC, Canada.
55. (2017, February 10). Smart* Data Set for Sustainability. Available online: http://traces.cs.umass.edu/index.php/Smart/Smart.
56. (2017, March 19). The Mnist Database. Available online: http://yann.lecun.com/exdb/mnist/.