FnnmOS-ELM: A Flexible Neural Network Mixed Online Sequential Elm
Source abstract
Abstract
The learning speed of online sequential extreme learning machine (OS-ELM) algorithms is much higher than that of convolutional neural networks (CNNs) or recurrent neural network (RNNs) on regression and simple classification datasets. However, the general feature extraction of OS-ELM makes it difficult to conveniently and effectively perform classification on some large and complex datasets, e.g., CIFAR. In this paper, we propose a flexible OS-ELM-mixed neural network, termed as fnnmOS-ELM. In this mixed structure, the OS-ELM can replace a part of fully connected layers in CNNs or RNNs. Our framework not only exploits the strong feature representation of CNNs or RNNs, but also performs at a fast speed in terms of classification. Additionally, it avoids the problem of long training time and large parameter size of CNNs or RNNs to some extent. Further, we propose a method for optimizing network performance by splicing OS-ELM after CNN or RNN structures. Iris, IMDb, CIFAR-10, and CIFAR-100 datasets are employed to verify the performance of the fnnmOS-ELM. The relationship between hyper-parameters and the performance of the fnnmOS-ELM is explored, which sheds light on the optimization of network performance. Finally, the experimental results demonstrate that the fnnmOS-ELM has a stronger feature representation and higher classification performance than contemporary methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and ELM · Domain Adaptation and Few-Shot Learning · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Engineering, Minzu University of China, Beijing 100081, China
State Key Laboratory for Turbulence and Complex System, Department of Mechanics and Engineering Science, BIC-ESAT, College of Engineering, Peking University, Beijing 100871, China
Funding
National Natural Science Foundation of China
References (55)
Show source-provided references
1. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
2. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
3. Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., and Torr, P.H.S. (2015, January 7–13). Conditional randomfields as recurrent neural networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.179 ↗
4. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and Lecun, Y. (2014, January 14–16). Overfeat: Integrated recognition, localization and detection using convolutional networks. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada.
5. Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. (2014, January 23–28). Deepface: Closing the gap to human level performance in face verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.220 ↗
6. Jozefowicz, R., Zaremba, W., and Sutskever, I. (2015, January 6–11). An empirical exploration of recurrent network architectures. Proceedings of the 32nd International Conference on Machine Learning, Lille, France.
7. Deep neural networks for acoustic modeling in speech recognition10.1109/MSP.2012.2205597 ↗
8. Efficient backprop
9. Introduction to multi-layer feed-forward neural networks10.1016/S0169-7439(97)00061-0 ↗
10. Huang, G.-B., Zhu, Q.-Y., and Siew, C.-K. (2004). Extreme Learning Machine: A New Learning Scheme of Feedforward Neural Networks. Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, Budapest, Hungary, 25–29 July 2004, IEEE.
11. Huang, G.-B., and Siew, C.K. (2004, January 6–9). Extreme learning machine: RBF network case. Proceedings of the 8th Control, Automation, Robotics and Vision Conference (ICARCV), Kunming, China.
12. Can threshold networks be trained directly?10.1109/TCSII.2005.857540 ↗
13. Extreme learning machine: Theory and applications10.1016/j.neucom.2005.12.126 ↗
14. Universal approximation using incremental constructive feedforward networks with random hidden nodes10.1109/TNN.2006.875977 ↗
15. A fast and accurate online sequential learning algorithm for feedforward networks10.1109/TNN.2006.880583 ↗
16. Deep Learning10.1038/nature14539 ↗
17. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
18. Xu, R., Tao, Y., Lu, Z., and Zhong, Y. (2018). Attention-Mechanism-Containing Neural Networks for High-Resolution Remote Sensing Image Classification. Remote Sens., 10.10.3390/rs10101602 ↗
19. Adaptation to New Microphones Using Artificial Neural Networks with Trainable Activation Functions
20. Predicting Infectious Disease Using Deep Learning and Big Data10.3390/ijerph15081596 ↗
21. Smooth function approximation using neural networks10.1109/TNN.2004.836233 ↗
22. Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1988). Neurocomputing: Foundations of Research. Learning Internal Representations by Error Propagation, MIT Press.10.1016/B978-1-4832-1446-7.50035-2 ↗
23. Geometrical interpretation and architecture selection of MLP10.1109/TNN.2004.836197 ↗
24. Classification ability of single hidden layer feedforward neural networks10.1109/72.846750 ↗
25. Generalized single-hidden layer feedforward networks for regression problems
26. Learning in single hidden-layer feedforward network models: Backpropagation in a spatial interaction modeling context10.1111/j.1538-4632.1996.tb00920.x ↗
27. Ngia, L.S.H., Sjoberg, J., and Viberg, M. (1998, January 1–4). Adaptive neural nets filter using a recursive Levenberg-Marquardt search direction. Proceedings of the Conference Record of the Thirty-Second Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA.
28. Caruana, R., Lawrence, S., and Giles, L. (December, January 27). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Proceedings of the 13th International Conference on Neural Information Processing Systems, Denver, CO, USA.
29. Learning capability and storage capacity of two-hidden-layer feedforward networks10.1109/TNN.2003.809401 ↗
30. Capabilities of a four-layered feedforward neural network: Four layers versus three10.1109/72.557662 ↗
31. Generalized inverse of matrices and its applications10.1080/00401706.1973.10489026 ↗
32. Sparse extreme learning machine for classification
33. Data partition learning with multiple extreme learningmachines
34. Sparse Bayesian extreme learning machine for multi-classification
35. Extreme learning machine for regression and multiclass classification10.1109/TSMCB.2011.2168604 ↗
36. Multiple kernel extreme learning machine10.1016/j.neucom.2013.09.072 ↗
37. Local receptive fields based extreme learning machine10.1109/MCI.2015.2405316 ↗
38. A hybrid deep learning CNN-ELM for age and gender classification10.1016/j.neucom.2017.08.062 ↗
39. Dropout: A simple way to prevent neural networks from overfitting
40. Blake, C.L. (2019, June 25). UCI Repository of Machine Learning Databases. Available online: http://archive.ics.uci.edu/ml/index.php.
41. New methods in iris recognition10.1109/TSMCB.2007.903540 ↗
42. Ahmed, A., Batagelj, V., Fu, X., Hong, S.-H., Merrick, D., and Mrvar, A. (2007, January 5–7). Visualisation and analysis of the internet movie database. Proceedings of the 6th International Asia-Pacific Symposium on Visualization, Sydney, Australia.10.1109/APVIS.2007.329304 ↗
43. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images, Department of Computer Science, University of Toronto. Tech. Rep. 001.
44. Ben-Hur, A., Horn, D., Siegelmann, H.T., and Vapnik, V. (2000, January 3–7). A support vector method for clustering. Proceedings of the 13th International Conference on Neural Information Processing Systems, Barcelona, Spain.
45. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., and Woo, W. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, MIT Press Cambridg.
46. Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11–13). Deep sparse rectifier neural networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, USA.
47. Term-weighting approaches in automatic text retrieval10.1016/0306-4573(88)90021-0 ↗
48. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., and Mei, Q. (2015, January 18–22). Line: Large-scale information network embedding. Proceedings of the 24th International Conference on World Wide Web, Florence, Italy.10.1145/2736277.2741093 ↗
49. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 11–14). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_38 ↗
50. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2016, January 2–4). Fast and accurate deep network learning by exponential linear units (ELUs). Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
51. Liang, M., and Hu, X. (2015, January 7–12). Recurrent convolutional neural network for object recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
52. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
53. Agostinelli, F., Hoffman, M.D., Sadowski, P.J., and Baldi, P. (2014). Learning activation functions to improve deep neural networks. arXiv.
54. Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P.T.P. (2016). On largebatch training for deep learning: Generalization gap and sharp minima. arXiv.
55. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, Lile, France.