Dynamic Depth Learning in Stacked AutoEncoders
Source abstract
Abstract
The effectiveness of deep learning models depends on their architecture and topology. Thus, it is essential to determine the optimal depth of the network. In this paper, we propose a novel approach to learn the optimal depth of a stacked AutoEncoder, called Dynamic Depth for Stacked AutoEncoders (DDSAE). DDSAE learns in an unsupervised manner the depth of a stacked AutoEncoder while training the network model. Specifically, we propose a novel objective function, aside from the AutoEncoder’s loss function to optimize the network depth: The optimization of the objective function determines the layers’ relevance weights. Additionally, we propose an algorithm that iteratively prunes the irrelevant layers based on the learned relevance weights. The performance of DDSAE was assessed using benchmark and real datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Video Surveillance and Tracking Methods · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia
Department of Information and Computer Science, College of Computer Science and Engineering, University of Hail, Hail 55476, Saudi Arabia
References (41)
Show source-provided references
1. A Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning10.1007/s11831-019-09344-w ↗
2. Machine Learning Paradigms for Speech Recognition: An Overview10.1109/TASL.2013.2244083 ↗
3. Al Handwritten Digit Recognition Using Machine Learning Algorithms10.17509/ijost.v3i1.10795 ↗
4. Progress in Neural NLP: Modeling, Learning, and Reasoning10.1016/j.eng.2019.12.014 ↗
5. Machine Learning in Computer Vision10.1016/j.procs.2020.03.355 ↗
6. Machine (Deep) Learning Methods for Image Processing and Radiomics10.1109/TRPMS.2019.2899538 ↗
7. Credit Card Fraud Detection Using Machine Learning Algorithms10.1016/j.procs.2020.01.057 ↗
8. Alom, M.Z., Taha, T.M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M.S., Hasan, M., Van Essen, B.C., Awwal, A.A.S., and Asari, V.K. (2019). A State-of-the-Art Survey on Deep Learning Theory and Architectures. Electronics, 8.10.3390/electronics8030292 ↗
9. (2021, March 08). Google AI. Available online: https://ai.google/tools/.
10. Apple (2021, March 08). Machine Learning Research. Available online: https://machinelearning.apple.com/.
11. (2021, March 08). Facebook AI. Available online: https://ai.facebook.com/.
12. (2021, March 08). Amazon AI. Available online: https://www.aboutamazon.com/news/amazon-ai.
13. Assessment of Autoencoder Architectures for Data Representation
14. Chicco, D., Sadowski, P., and Baldi, P. (2014, January 20–23). Deep Autoencoder Neural Networks for Gene Ontology Annotation Predictions. Proceedings of the ACM BCB 2014—5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics, Newport Beach, CA, USA.10.1145/2649387.2649442 ↗
15. Alom, M.Z., and Taha, T.M. (2017, January 27–30). Network Intrusion Detection for Cyber Security Using Unsupervised Deep Learning Approaches. Proceedings of the IEEE National Aerospace and Electronics Conference (NAECON), Dayton, OH, USA.10.1109/NAECON.2017.8268746 ↗
16. A Survey of Autoencoder-Based Recommender Systems10.1007/s11704-018-8052-6 ↗
17. Stacked Autoencoders for the P300 Component Detection10.3389/fnins.2017.00302 ↗
18. Evolving Deep Convolutional Variational Autoencoders for Image Classification10.1109/TEVC.2020.3047220 ↗
19. Souza, P.V.d.C., Torres, L.C.B., Silva, G.R.L., de Braga, A.P., and Lughofer, E. (2020). An Advanced Pruning Method in the Architecture of Extreme Learning Machines Using L1-Regularization and Bootstrapping. Electronics, 9.10.3390/electronics9050811 ↗
20. Hyperspectral Unmixing Using a Neural Network Autoencoder10.1109/ACCESS.2018.2818280 ↗
21. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
22. Zhai, J., Zhang, S., Chen, J., and He, Q. (2018, January 7–10). Autoencoder and Its Various Variants. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.10.1109/SMC.2018.00080 ↗
23. A Stacked Autoencoder-Based Deep Neural Network for Achieving Gearbox Fault Diagnosis
24. Gogoi, M., and Begum, S.A. (2017, January 14–16). Image Classification Using Deep Autoencoders. Proceedings of the IEEE International Conference on Computational Intelligence and Computing Research, ICCIC 2017, Coimbatore, India.10.1109/ICCIC.2017.8524276 ↗
25. Recognition of Geochemical Anomalies Using a Deep Autoencoder Network10.1016/j.cageo.2015.10.006 ↗
26. Hajewski, J., and Oliveira, S. (2020, January 6–8). An Evolutionary Approach to Variational Autoencoders. Proceedings of the 10th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC47524.2020.9031239 ↗
27. Charte, F., Rivera, A.J., Martínez, F., and del Jesus, M.J. (June, January 31). Automating Autoencoder Architecture Configuration: An Evolutionary Approach. Proceedings of the International Work-Conference on the Interplay Between Natural and Artificial Computation, Almería, Spain.
28. Lander, S., and Shang, Y. (2015, January 1–5). EvoAE-A New Evolutionary Method for Training Autoencoders for Deep Learning Networks. Proceedings of the IEEE 39th Annual Computer Software and Applications Conference, Taichung, Taiwan.10.1109/COMPSAC.2015.63 ↗
29. Online Incremental Feature Learning with Denoising Autoencoders
30. DEVDAN: Deep Evolving Denoising Autoencoder10.1016/j.neucom.2019.07.106 ↗
31. Stacked Pruning Sparse Denoising Autoencoder Based Intelligent Fault Diagnosis of Rolling Bearings10.1016/j.asoc.2019.106060 ↗
32. Ghosh, A., and Tsutsui, S. (2003). Advances in Evolutionary Computing: Theory and Applications, Springer.10.1007/978-3-642-18965-4 ↗
33. Sun, S., Chen, W., Wang, L., Liu, X., and Liu, T.Y. (2016, January 12–17). On the Depth of Deep Neural Networks: A Theoretical View. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10243 ↗
34. Saul, L., Weinberger, K., Ham, J., and Sha, F. (2006). Semisupervised Learning, MIT Press.
35. Yann, L., Corinna, C., and Christopher, B. (1998). The Mnist Database of Handwritten Digits, The Courant Institute of Mathematical Sciences.
36. Krizhevsky, A., Nair, V., and Hinton, G. (2009). CIFAR-10 and CIFAR-100 Datasets, University of Toronto. Technical Report.
37. A Comparative Analysis of Speech Signal Processing Algorithms for Parkinson’s Disease Classification and the Use of the Tunable Q-Factor Wavelet Transform10.1016/j.asoc.2018.10.022 ↗
38. UCI (2023, April 24). Machine Learning Repository: Parkinson’s Disease Classification Data Set. Available online: https://archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification.
39. Patrício, M., Pereira, J., Crisóstomo, J., Matafome, P., Gomes, M., Seiça, R., and Caramelo, F. (2018). Using Resistin, Glucose, Age and BMI to Predict the Presence of Breast Cancer. BMC Cancer, 18.10.1186/s12885-017-3877-1 ↗
40. UCI (2023, April 24). Machine Learning Repository: Breast Cancer Coimbra Data Set. Available online: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Coimbra.
41. Magnus, E. (2021). Learning Deep Learning: Theory and Practice of Neural Networks, Computer Vision, NLP, and Transformers Using TensorFlow, Deep Learning Institute. [1st ed.].