Comparing Vision Transformers and Convolutional Neural Networks for Image Classification: A Literature Review
Source abstract
Abstract
Transformers are models that implement a mechanism of self-attention, individually weighting the importance of each part of the input data. Their use in image classification tasks is still somewhat limited since researchers have so far chosen Convolutional Neural Networks for image classification and transformers were more targeted to Natural Language Processing (NLP) tasks. Therefore, this paper presents a literature review that shows the differences between Vision Transformers (ViT) and Convolutional Neural Networks. The state of the art that used the two architectures for image classification was reviewed and an attempt was made to understand what factors may influence the performance of the two deep learning architectures based on the datasets used, image size, number of target classes (for the classification problems), hardware, and evaluated architectures and top results. The objective of this work is to identify which of the architectures is the best for image classification and under what conditions. This paper also describes the importance of the Multi-Head Attention mechanism for improving the performance of ViT in image classification.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: COVID-19 diagnosis using AI · Anomaly Detection Techniques and Applications · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Polytechnic of Coimbra, Coimbra Institute of Engineering (ISEC), Rua Pedro Nunes, 3030-199 Coimbra, Portugal
References (30)
Show source-provided references
1. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
2. Saha, S. (2023, January 08). A Comprehensive Guide to Convolutional Neural Networks—The ELI5 Way. Available online: https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53.
3. Literature Review as a Research Methodology: An Overview and Guidelines10.1016/j.jbusres.2019.07.039 ↗
4. Software Defect Prediction Using Ensemble Learning: A Systematic Literature Review10.1109/ACCESS.2021.3095559 ↗
5. Benz, P., Ham, S., Zhang, C., Karjauv, A., and Kweon, I.S. (2021). Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs. arXiv.
6. Bai, Y., Mei, J., Yuille, A., and Xie, C. (2021). Are Transformers More Robust Than CNNs?. arXiv.
7. Tyagi, K., Pathak, G., Nijhawan, R., and Mittal, A. (2021, January 2). Detecting Pneumonia Using Vision Transformer and Comparing with Other Techniques. Proceedings of the 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA), IEEE, Coimbatore, India.10.1109/ICECA52323.2021.9676146 ↗
8. Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. (2021). Do Vision Transformers See Like Convolutional Neural Networks?. arXiv.
9. Gheflati, B., and Rivaz, H. (2021). Vision Transformer for Classification of Breast Ultrasound Images. arXiv.10.1109/EMBC48229.2022.9871809 ↗
10. Zhou, H.-Y., Lu, C., Yang, S., and Yu, Y. (2021, January 17). ConvNets vs. Transformers: Whose Visual Representations Are More Transferable?. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), IEEE, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00252 ↗
11. A Vision Transformer for Emphysema Classification Using CT Images10.1088/1361-6560/ac3dc8 ↗
12. Comparing Vision Transformers and Convolutional Nets for Safety Critical Systems
13. Galdran, A., Carneiro, G., and Ballester, M.A.G. (2022). Convolutional Nets Versus Vision Transformers for Diabetic Foot Ulcer Classification. arXiv.10.1007/978-3-030-94907-5_2 ↗
14. Cuenat, S., and Couturier, R. (2022, January 18). Convolutional Neural Network (CNN) vs Vision Transformer (ViT) for Digital Holography. Proceedings of the 2022 2nd International Conference on Computer, Control and Robotics (ICCCR), IEEE, Shanghai, China.10.1109/ICCCR54399.2022.9790134 ↗
15. Coccomini, D.A., Caldelli, R., Falchi, F., Gennaro, C., and Amato, G. (2022, January 27–30). Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection. Proceedings of the 1st International Workshop on Multimedia AI against Disinformation, Newark, NJ, USA.10.1145/3512732.3533582 ↗
16. Wang, H. (2022, January 27–29). Traffic Sign Recognition with Vision Transformers. Proceedings of the 6th International Conference on Information System and Data Mining, Silicon Valley, CA, USA.10.1145/3546157.3546166 ↗
17. An Improved Transformer Network for Skin Cancer Classification10.1016/j.compbiomed.2022.105939 ↗
18. CNN and Transformer Framework for Insect Pest Classification10.1016/j.ecoinf.2022.101846 ↗
19. Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead10.1016/j.neunet.2022.06.038 ↗
20. Vision Transformer-Based Autonomous Crack Detection on Asphalt and Concrete Surfaces10.1016/j.autcon.2022.104316 ↗
21. Reedha, R., Dericquebourg, E., Canals, R., and Hafiane, A. (2022). Vision Transformers for Weeds and Crops Classification of High Resolution UAV Images. Remote Sens., 14.10.3390/rs14030592 ↗
22. Platt, J., Koller, D., Singer, Y., and Roweis, S. (2007). Advances in Neural Information Processing Systems, Curran Associates, Inc.
23. Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2020). Sharpness-Aware Minimization for Efficiently Improving Generalization. arXiv.
24. The Extragradient Method for Finding Saddle Points and Other Problems
25. Dataset of Breast Ultrasound Images10.1016/j.dib.2019.104863 ↗
26. Automated Breast Ultrasound Lesions Detection Using Convolutional Neural Networks10.1109/JBHI.2017.2731873 ↗
27. Zhang, R. (2019). Making Convolutional Networks Shift-Invariant Again. arXiv.
28. Attention Is All You Need
29. Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., and Feng, J. (2021). DeepViT: Towards Deeper Vision Transformer. arXiv.
30. Amorim, J.P., Domingues, I., Abreu, P.H., and Santos, J.A.M. (2018, January 25–27). Interpreting Deep Learning Models for Ordinal Problems. Proceedings of the European Symposium on Artificial Neural Networks, Bruges, Belgium.