Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
Source abstract
Abstract
In this paper, we propose two novel Adaptive Neural Network Approaches (ANNAs), which are intended to automatically learn the optimal network depth. In particular, the proposed class-independent and class-dependent ANNAs address two main challenges faced by typical deep learning paradigms. Namely, they overcome the problems of setting the optimal network depth and improving the model interpretability. Specifically, ANNA approaches simultaneously train the network model, learn the network depth in an unsupervised manner, and assign fuzzy relevance weights to each network layer to better decipher the model behavior. In addition, two novel cost functions were designed in order to optimize the layer fuzzy relevance weights along with the model hyper-parameters. The proposed ANNA approaches were assessed using standard benchmarking datasets and performance measures. The experiments proved their effectiveness compared to typical deep learning approaches, which rely on empirical tuning and scaling of the network depth. Moreover, the experimental findings demonstrated the ability of the proposed class-independent and class-dependent ANNAs to decrease the network complexity and build lightweight models for less overfitting risk and better generalization.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Human Pose and Action Recognition · Image Enhancement Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh 11362, Saudi Arabia
References (44)
Show source-provided references
1. Computer vision10.1126/science.224.4654.1225 ↗
2. Sonka, M., Hlavac, V., and Boyle, R. (2014). Image Processing, Analysis, and Machine Vision, Cengage Learning. Available online: https://books.google.com/books?hl=en&lr=&id=DcETCgAAQBAJ&oi=fnd&pg=PR11&ots=ynj1Cr2sqH&sig=dEZoUJLqh6cK7ptmT18zu6gAc_k.
3. Pattern Recognition and Machine Learning10.18637/jss.v017.b05 ↗
4. A survey of deep neural network architectures and their applications10.1016/j.neucom.2016.12.038 ↗
5. Machine learning in automatic speech recognition: A survey10.1080/02564602.2015.1010611 ↗
6. Jaitly, N., and Hinton, G. (2011, January 22–27). Learning a better representation of speech soundwaves using restricted boltzmann machines. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.10.1109/ICASSP.2011.5947700 ↗
7. Mohamed, A.R., Yu, D., and Deng, L. (2010, January 26–30). Investigation of full-sequence training of deep belief networks for speech recognition. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH, Chiba, Japan.10.21437/Interspeech.2010-304 ↗
8. Alom, M.Z., Taha, T.M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M.S., Hasan, M., Van Essen, B.C., Awwal, A.A., and Asari, V.K. (2019). A State-of-the-Art Survey on Deep Learning Theory and Architectures. Electronics, 8.10.3390/electronics8030292 ↗
9. Deep Learning
10. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
11. Convolution Neural Network for Medical Image Pattern Recognition10.1016/0893-6080(95)00061-5 ↗
12. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
13. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., and Chua, T.S. (2017, January 21–26). SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA.10.1109/CVPR.2017.667 ↗
14. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Li, F.F. (2014, January 23–28). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
15. Yang, J.B., Nguyen, M.N., San, P.P., Li, X.L., and Krishnaswamy, S. (2015, January 25–31). Deep convolutional neural networks on multichannel time series for human activity recognition. Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
16. Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks10.1109/TETCI.2017.2784878 ↗
17. Xu, Y., Kong, Q., Huang, Q., Wang, W., and Plumbley, M.D. (2017, January 19–25). Convolutional gated recurrent neural network incorporating spatial features for audio tagging. Proceedings of the International Joint Conference on Neural Networks, Melbourne, Australia.10.1109/IJCNN.2017.7966291 ↗
18. Semi-Supervised Locality Preserving Dense Graph Neural Network with ARMA Filters and Context-Aware Learning for Hyperspectral Image Classification
19. Multi-feature fusion: Graph neural network and CNN combining for hyperspectral image classification10.1016/j.neucom.2022.06.031 ↗
20. Self-Supervised Locality Preserving Low-Pass Graph Convolutional Embedding for Large-Scale Hyperspectral Image Clustering
21. Unsupervised Self-correlated Learning Smoothy Enhanced Locality Preserving Graph Convolution Embedding Clustering for Hyperspectral Images
22. Review of deep learning algorithms and architectures10.1109/ACCESS.2019.2912200 ↗
23. Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M., and Yang, S. (2017, January 6–11). AdaNet: Adaptive structural learning of artificial neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML 2017), Sydney, Australia.
24. Sun, S., Chen, W., Wang, L., Liu, X., and Liu, T.Y. (2016, January 12–17). On the depth of deep neural networks: A theoretical view. Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI 2016, Phoenix, AZ, USA. Available online: http://arxiv.org/abs/1506.05232.10.1609/aaai.v30i1.10243 ↗
25. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
26. Algorithms for Hyper-Parameter Optimization
27. GA-TSKfnn: Parameters tuning of fuzzy neural network using genetic algorithms10.1016/j.eswa.2005.06.001 ↗
28. Optimal parameters selection for BP neural network based on particle swarm optimization: A case study of wind speed forecasting10.1016/j.knosys.2013.11.015 ↗
29. Online incremental feature learning with denoising autoencoders
30. Philipp, G., and Carbonell, J.G. (2017, January 24–26). Nonparametric Neural Networks. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France. Available online: http://arxiv.org/abs/1712.05440.
31. Draelos, T.J., Miner, N.E., Lamb, C.C., Cox, J.A., Vineyard, C.M., Carlson, K.D., Severa, W.M., James, C.D., and Aimone, J.B. (December, January 27). Neurogenesis Deep Learning. Proceedings of the International Joint Conference on Neural Networks, Perth, Australia.
32. Yoon, J., Yang, E., Lee, J., and Hwang, S.J. (May, January 30). Lifelong Learning with Dynamically Expandable Networks. Proceedings of the 6th International Conference on Learning Representations. (ICLR 2018), Vancouver, BC, Canada. Available online: http://arxiv.org/abs/1708.01547.
33. Learning the number of neurons in deep networks
34. Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. (2017, January 24–26). Pruning convolutional neural networks for resource efficient inference. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
35. Training very deep networks
36. Deep residual learning for image recognition
37. Deep networks with stochastic depth
38. Wang, J., Wei, Z., Zhang, T., and Zeng, W. (2016). Deeply-Fused Nets. arXiv.
39. Learning deep architectures for AI10.1561/2200000006 ↗
40. Attention-Based Adaptive Spectral-Spatial Kernel ResNet for Hyperspectral Image Classification10.1109/TGRS.2020.3043267 ↗
41. Lagrange Multipliers10.1201/9781420000269.axg ↗
42. LeCun, Y., and Cortes, C. (2021, June 03). MNIST Handwritten Digit Database. AT&T Labs. Available online: http://yann.lecun.com/exdb/mnist/.
43. Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report.
44. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015—Conference Track Proceedings, San Diego, CA, USA. Available online: http://www.robots.ox.ac.uk/.