Deep Residual Learning for Image Recognition: A Survey
Source abstract
Abstract
Deep Residual Networks have recently been shown to significantly improve the performance of neural networks trained on ImageNet, with results beating all previous methods on this dataset by large margins in the image classification task. However, the meaning of these impressive numbers and their implications for future research are not fully understood yet. In this survey, we will try to explain what Deep Residual Networks are, how they achieve their excellent results, and why their successful implementation in practice represents a significant advance over existing techniques. We also discuss some open questions related to residual learning as well as possible applications of Deep Residual Networks beyond ImageNet. Finally, we discuss some issues that still need to be resolved before deep residual learning can be applied on more complex problems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · COVID-19 diagnosis using AI · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou 510006, China
Department of New Networks, Peng Cheng Laboratory, Shenzhen 518055, China
Department of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Funding
Major Key Project of PCL · National Natural Science Foundation of China · Guangzhou Science and technology planning project · Guangdong Higher Education Innovation Group · Guangzhou Higher Education Innovation Group
References (94)
Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
2. A general framework for maximizing likelihood under incomplete data10.1016/j.ijar.2017.10.030 ↗
3. A hybrid quantum–classical neural network with deep residual learning10.1016/j.neunet.2021.05.028 ↗
4. Feng, Z., Nie, D., Wang, L., and Shen, D. (2018, January 4–7). Semi-supervised learning for pelvic MR image segmentation based on multi-task residual fully convolutional networks. Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.10.1109/ISBI.2018.8363713 ↗
5. Few-shot contrastive learning for image classification and its application to insulator identification10.1007/s10489-021-02769-6 ↗
6. Classification of Trash for Recyclability Status
7. Cascaded deep learning classifiers for computer-aided diagnosis of COVID-19 and pneumonia diseases in X-ray scans10.1007/s40747-020-00199-4 ↗
8. A Hybrid CNN–LSTM Network for the Classification of Human Activities Based on Micro-Doppler Radar10.1109/ACCESS.2020.2971064 ↗
9. (2017). FPGA Acceleration of Convolutional Neural Networks, Nallatech.
10. Classification model of ‘Toraja’ arabica coffee fruit ripeness levels using convolution neural network approach10.33096/ilkom.v13i3.861.226-234 ↗
11. Automated System for Chromosome Karyotyping to Recognize the Most Common Numerical Abnormalities Using Deep Learning10.1109/ACCESS.2020.3019937 ↗
12. Avtar, R., Tripathi, S., Aggarwal, A.K., and Kumar, P. (2019). Population–Urbanization–Energy Nexus: A Review. Resources, 8.10.3390/resources8030136 ↗
13. Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC
14. Akhand, M., Roy, S., Siddique, N., Kamal, A.S., and Shimamura, T. (2021). Facial Emotion Recognition Using Transfer Learning in the Deep CNN. Electronics, 10.10.3390/electronics10091036 ↗
15. Effects of image compression on face image manipulation detection: A case study on facial retouching10.1049/bme2.12027 ↗
16. Siam, M., Elkerdawy, S., Jagersand, M., and Yogamani, S. (2017, January 16–19). Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges. Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan.10.1109/ITSC.2017.8317714 ↗
17. Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., and Timofte, R. (IEEE Trans. Pattern Anal. Mach. Intell., 2021). Plug-and-Play Image Restoration with Deep Denoiser Prior, IEEE Trans. Pattern Anal. Mach. Intell., early access.10.1109/TPAMI.2021.3088914 ↗
18. Sangeetha, V., and Prasad, K.J.R. (2006). Deep Residual Learning for Image Recognition Kaiming. Indian J. Chem.-Sect. B Org. Med. Chem.
19. An Adaptive and Asymmetric Residual Hash for Fast Image Retrieval10.1109/ACCESS.2019.2922738 ↗
20. “Never fry carrots without chopping” Generating Cooking Recipes from Cooking Videos Using Deep Learning Considering Previous Process10.2991/ijndc.k.190710.002 ↗
21. Avtar, R., Sahu, N., Aggarwal, A.K., Chakraborty, S., Kharrazi, A., Yunus, A.P., Dou, J., and Kurniawan, T.A. (2019). Exploring Renewable Energy Resources Using Remote Sensing and GIS—A Review. Resources, 8.10.3390/resources8030149 ↗
22. Assessing sustainable development prospects through remote sensing: A review
23. A residual dense network assisted sparse view reconstruction for breast computed tomography10.1038/s41598-020-77923-0 ↗
24. Deep learning based spectral CT imaging10.1016/j.neunet.2021.08.026 ↗
25. Jalali, Y., Fateh, M., Rezvani, M., Abolghasemi, V., and Anisi, M.H. (2021). ResBCDU-Net: A Deep Learning Framework for Lung CT Image Segmentation. Sensors, 21.10.3390/s21010268 ↗
26. Lung CT Image Recognition using Deep Learning Techniques to Detect Lung Cancer10.30534/ijeter/2020/113872020 ↗
27. LiteDepthwiseNet: A Lightweight Network for Hyperspectral Image Classification
28. Jafar, A., and Myungho, L. (2020, January 17–21). Hyperparameter Optimization for Deep Residual Learning in Image Classification. Proceedings of the 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C), Washington, DC, USA.10.1109/ACSOS-C51401.2020.00024 ↗
29. Very Deep Convolutional Neural Networks for Noise Robust Speech Recognition10.1109/TASLP.2016.2602884 ↗
30. Training Very Deep CNNs for General Non-Blind Deconvolution10.1109/TIP.2018.2815084 ↗
31. Tai, Y., Yang, J., and Liu, X. (2017, January 21–26). Image super-resolution via deep recursive residual network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.298 ↗
32. Matching Software-Generated Sketches to Face Photographs With a Very Deep CNN, Morphed Faces, and Transfer Learning10.1109/TIFS.2017.2788002 ↗
33. Moriya, S., and Shibata, C. (2018, January 23–27). Transfer Learning Method for Very Deep CNN for Text Classification and Methods for its Evaluation. Proceedings of the 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Tokyo, Japan.10.1109/COMPSAC.2018.10220 ↗
34. Afzal, M.Z., Kolsch, A., Ahmed, S., and Liwicki, M. (2017, January 9–15). Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.10.1109/ICDAR.2017.149 ↗
35. A comprehensive review of deep learning-based single image super-resolution10.7717/peerj-cs.621 ↗
36. A Power-Efficient Optimizing Framework FPGA Accelerator Based on Winograd for YOLO10.1109/ACCESS.2020.2995330 ↗
37. Lim, H.K., Kim, J.B., Heo, J.S., Kim, K., Hong, Y.G., and Han, Y.H. (2019, January 11–13). Packet-based network traffic classification using deep learning. Proceedings of the 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), Okinawa, Japan.10.1109/ICAIIC.2019.8669045 ↗
38. (2022, September 02). Available online: https://cyberleninka.ru/article/n/reshenie-zadach-vychislitelnoy-gidrodinamiki-s-primeneniem-tehnologii-nvidia-cuda-articlehead-tehnologiya-nvidia-cuda-v-zadachah/viewer.
39. NVIDIA (2015). Cuda C Best Practices Guide, Nvidia Corp.
40. Yasin, S., Iqbal, N., Ali, T., Draz, U., Alqahtani, A., Irfan, M., Rehman, A., Glowacz, A., Alqhtani, S., and Proniewska, K. (2021). Severity Grading and Early Retinopathy Lesion Detection through Hybrid Inception-ResNet Architecture. Sensors, 21.10.3390/s21206933 ↗
41. VBSF: A new storage format for SIMD sparse matrix–vector multiplication on modern processors10.1007/s11227-019-02835-4 ↗
42. Quantum Supremacy Circuit Simulation on Sunway TaihuLight10.1109/TPDS.2019.2947511 ↗
43. Trailblazers in Electromechanical Computing [Historical]10.1109/MIE.2017.2694578 ↗
44. Li, Y., and Chen, H. (2021, January 14–16). Image recognition based on deep residual shrinkage Network. Proceedings of the 2021 International Conference on Artificial Intelligence and Electromechanical Automation (AIEA), Guangzhou, China.10.1109/AIEA53260.2021.00077 ↗
45. Yang, Z., Wu, B., Wang, Z., Li, Y., and Feng, H. (SSRN Electron. J., 2022). Image Recognition Based on an Improved Deep Residual Shrinkage Network, SSRN Electron. J., in press.10.2139/ssrn.4013383 ↗
46. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
47. Anomaly Detection in Automated Vehicles Using Multistage Attention-Based Convolutional Neural Network10.1109/TITS.2020.3025875 ↗
48. EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks10.1109/TIP.2019.2937724 ↗
49. Krizhevsky, A., Nair, V., and Hinton, G. (2022, August 08). CIFAR-10 and CIFAR-100 Datasets. Available online: https://www.cs.toronto.edu/~kriz/cifar.html.
50. Deep learning for COVID-19 chest CT (computed tomography) image analysis: A lesson from lung cancer10.1016/j.csbj.2021.02.016 ↗
51. Remote Sensing Data Augmentation through Adversarial Training10.1109/JSTARS.2021.3110842 ↗
52. Efficient clustering for aggregate loads: An unsupervised pretraining based method10.1016/j.energy.2020.118617 ↗
53. Riviere, M., Joulin, A., Mazare, P.-E., and Dupoux, E. (2020, January 4–8). Unsupervised Pretraining Transfers Well Across Languages. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9054548 ↗
54. A Novel Hybrid Deep Learning Model for Sentiment Classification10.1109/ACCESS.2020.2982538 ↗
55. Android Malware Detection Based on a Hybrid Deep Learning Model10.1155/2020/8863617 ↗
56. A comprehensive survey of AI-enabled phishing attacks detection techniques10.1007/s11235-020-00733-2 ↗
57. FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search10.1109/TPAMI.2020.3044416 ↗
58. A Comprehensive Survey on Graph Neural Networks10.1109/TNNLS.2020.2978386 ↗
59. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K.Q. (2016). Deep networks with stochastic depth. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer.10.1007/978-3-319-46493-0_39 ↗
60. Chen, D., Zhang, W., Xu, X., and Xing, X. (2016, January 13–16). Deep networks with stochastic depth for acoustic modelling. Proceedings of the 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), Jeju, Korea.10.1109/APSIPA.2016.7820692 ↗
61. Koonce, B. (2021). SqueezeNet. Convolutional Neural Networks with Swift for Tensorflow, Apress.10.1007/978-1-4842-6168-2 ↗
62. Bobenko, A.I., Lutz, C.O.R., Pottmann, H., and Techter, J. (2021). Checkerboard Incircular Nets. SpringerBriefs in Mathematics, Springer.10.1007/978-3-030-81847-0_8 ↗
63. A fully automatic deep learning system for COVID-19 diagnostic and prognostic analysis10.1183/13993003.00775-2020 ↗
64. Kumar, D., Taylor, G.W., and Wong, A. (2017). Opening the Black Box of Financial AI with CLEAR-Trade: A CLass-Enhanced Attentive Response Approach for Explaining and Visualizing Deep Learning-Driven Stock Market Prediction. J. Comput. Vis. Imaging Syst., 3.10.15353/vsnl.v3i1.166 ↗
65. Pest identification via deep residual learning in complex background10.1016/j.compag.2017.08.005 ↗
66. He, S., Jonsson, E., Mader, C.A., and Martins, J.R.R.A. (2019, January 7–11). Aerodynamic Shape Optimization with Time Spectral Flutter Adjoint. Proceedings of the AIAA Scitech 2019 Forum, San Diego, CA, USA.10.2514/6.2019-0697 ↗
67. Deep residual learning for image steganalysis10.1007/s11042-017-4440-4 ↗
68. Bearing Fault Detection Using Scalogram and Switchable Normalization-Based CNN (SN-CNN)10.1109/ACCESS.2021.3089698 ↗
69. Allegra, A., Tonacci, A., Sciaccotta, R., Genovese, S., Musolino, C., Pioggia, G., and Gangemi, S. (2022). Machine Learning and Deep Learning Applications in Multiple Myeloma Diagnosis, Prognosis, and Treatment Selection. Cancers, 14.10.3390/cancers14030606 ↗
70. Star–galaxy classification using deep convolutional neural networks10.1093/mnras/stw2672 ↗
71. Large-scale distributed L-BFGS10.1186/s40537-017-0084-5 ↗
72. Word2Vec10.1017/S1351324916000334 ↗
73. Data mining and machine learning methods for sustainable smart cities traffic classification: A survey10.1016/j.scs.2020.102177 ↗
74. IoT malicious traffic identification using wrapper-based feature selection mechanisms10.1016/j.cose.2020.101863 ↗
75. CorrAUC: A Malicious Bot-IoT Traffic Detection Method in IoT Network Using Machine-Learning Techniques10.1109/JIOT.2020.3002255 ↗
76. A prospective randomized trial examining the use of a closed suction drain shows no influence on strength or function in primary total knee arthroplasty10.1302/0301-620X.101B7.BJJ-2018-1420.R1 ↗
77. Adaptive Weight Decay for Deep Neural Networks10.1109/ACCESS.2019.2937139 ↗
78. Blockchain-Based Automated System for Identification and Storage of Networks10.1155/2021/6694281 ↗
79. Recent advances in convolutional neural networks10.1016/j.patcog.2017.10.013 ↗
80. Buolamwini, J., and Gebru, T. (2018, January 23–24). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the Conference on Fairness, Accountability and Transparency, PMLR, New York, NY, USA.
81. Datta, A., and Swamidass, S. (2022, January 3–5). Fair-Net: A Network Architecture for Reducing Performance Disparity between Identifiable Sub-populations. Proceedings of the 14th International Conference on Agents and Artificial Intelligence, Online.10.5220/0010877400003116 ↗
82. Kim, M.P., Ghorbani, A., and Zou, J. (2019, January 27–28). Multiaccuracy: Black-box post-processing for fairness in classification. Proceedings of the AIES 2019—2019 AAAI/ACM Conference on AI, Ethics, and Society, Honolulu, HI, USA.10.1145/3306618.3314287 ↗
83. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6–11). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, Australia.
84. Datta, A., Flynn, N.R., and Swamidass, S.J. (2021, January 18–22). Cal-Net: Jointly Learning Classification and Calibration on Imbalanced Binary Classification Tasks. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534411 ↗
85. Adaptive Batch Normalization for practical domain adaptation10.1016/j.patcog.2018.03.005 ↗
86. Singh, A.K., Kumar, A., Mahmud, M., Kaiser, M.S., and Kishore, A. (2021). COVID-19 Infection Detection from Chest X-Ray Images Using Hybrid Social Group Optimization and Support Vector Classifier. Cogn. Comput.10.1007/s12559-021-09848-3 ↗
87. Sik-Ho, T. (Medium Note, 2018). Review: AlexNet, CaffeNet—Winner of ILSVRC 2012 (Image Classification), Medium Note.
88. Classification of lymphocytes, monocytes, eosinophils, and neutrophils on white blood cells using hybrid Alexnet-GoogleNet-SVM10.1007/s42452-021-04485-9 ↗
89. Multi-level residual network VGGNet for fish species classification
90. Zhou, T., Zhao, Y., and Wu, J. (2021, January 19–22). ResNeXt and Res2Net Structures for Speaker Verification. Proceedings of the 2021 IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383531 ↗
91. Experiences of Internet traffic monitoring with tstat10.1109/MNET.2011.5772055 ↗
92. Multi-classification approaches for classifying mobile app traffic10.1016/j.jnca.2017.11.007 ↗
93. Feng, H., Misra, V., and Rubenstein, D. (2007). The CIFAR-10 dataset. Electr. Eng., 35.10.1145/1269899.1254906 ↗
94. Stanford Vision Lab. (2016). ImageNet Dataset, Stanford Vision Lab, Stanford University.