A Comprehensive Survey of Recent Transformers in Image, Video and Diffusion Models
Source abstract
Abstract
Transformer models have emerged as dominant networks for various tasks in computer vision compared to Convolutional Neural Networks (CNNs). The transformers demonstrate the ability to model long-range dependencies by utilizing a self-attention mechanism. This study aims to provide a comprehensive survey of recent transformer-based approaches in image and video applications, as well as diffusion models. We begin by discussing existing surveys of vision transformers and comparing them to this work. Then, we review the main components of a vanilla transformer network, including the self-attention mechanism, feed-forward network, position encoding, etc. In the main part of this survey, we review recent transformer-based models in three categories: Transformer for downstream tasks, Vision Transformer for Generation, and Vision Transformer for Segmentation. We also provide a comprehensive overview of recent transformer models for video tasks and diffusion models. We compare the performance of various hierarchical transformer networks for multiple tasks on popular benchmark datasets. Finally, we explore some future research directions to further improve the field.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Attention is all you need
2. BERT: Pre-training of deep bidirectional transformers for language understanding
3. Language models are unsupervised multitask learners
4. Language models are few-shot learners
5. Imagenet classification with deep convolutional neural networks
6. Non-local neural networks
7. Squeeze-and-excitation networks
8. Orthogonal convolutional neural networks
9. CBAM: Convolutional block attention module
10. An image is worth 16 ×16 words: Transformers for image recognition at scale
11. Transformers in vision: A survey10.1145/3505244 ↗
12. A survey of visual transformers
13. A. M. Hafiz, S. A. Parah, and R. U. A. Bhat, “Attention mechanisms and deep learning for machine vision: A survey of the state of the art,” arXiv preprint arXiv:2106.07550, 2021.10.21203/rs.3.rs-510910/v1 ↗
14. A survey of transformers10.1016/j.aiopen.2022.10.001 ↗
15. Are we ready for a new paradigm shift? A survey on visual deep MLP10.1016/j.patter.2022.100520 ↗
16. Video transformers: A survey10.1109/TPAMI.2023.3243465 ↗
17. E. Min et al., “Transformer for graphs: An overview from architecture perspective,” arXiv preprint arXiv:2202.08455, 2022.
18. Survey: Transformer based video-language pre-training10.1016/j.aiopen.2022.01.001 ↗
19. A survey on vision transformer10.1109/TPAMI.2022.3152247 ↗
20. Y. Yang et al., “Transformers meet visual learning understanding: A comprehensive review,” arXiv preprint arXiv:2203.12944, 2022.
21. K. Islam, “Recent advances in vision transformer: A survey and outlook of recent work,” arXiv preprint arXiv:2203.01536, 2022.
22. Transformers in computational visual media: A survey10.1007/s41095-021-0247-3 ↗
23. MLP-Mixer: An all-MLP architecture for vision
24. ResMLP: Feedforward networks for image classification with data-efficient training10.1109/TPAMI.2022.3206148 ↗
25. L. Melas-Kyriazi, “Do you even need attention? A stack of feed-forward layers does surprisingly well on imagenet,” arXiv preprint arXiv:2105.02723, 2021.
26. Deep residual learning for image recognition
27. J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
28. Emerging properties in self-supervised vision transformers
29. MSG-Transformer: Exchanging local spatial information by manipulating messenger tokens
30. Pyramid Vision Transformer: A versatile backbone for dense prediction without convolutions
31. PVT v2: Improved baselines with pyramid vision transformer10.1007/s41095-022-0274-8 ↗
32. Swin transformer: Hierarchical vision transformer using shifted windows
33. Swin Transformer V2: Scaling up capacity and resolution
34. Attention-based residual autoencoder for video anomaly detection10.1007/s10489-022-03613-1 ↗
35. Uformer: A general U-shaped transformer for image restoration
36. Y. Li, K. Zhang, J. Cao, R. Timofte, and L. van Gool, “LocalViT: Bringing locality to vision transformers,” arXiv preprint arXiv:2104.05707, 2021.
37. Incorporating convolution designs into visual transformers
38. Restormer: Efficient transformer for high-resolution image restoration
39. Twins: Revisiting the design of spatial attention in vision transformers
40. Cswin transformer: A general vision transformer backbone with cross-shaped windows
41. Z. Huang, Y. Ben, G. Luo, P. Cheng, G. Yu and B. Fu, “Shuffle transformer: Rethinking spatial shuffle for vision transformer,” arXiv preprint arXiv:2106.03650, 2021.
42. Glance-and-Gaze vision transformer
43. Neighborhood attention transformer
44. DaViT: Dual attention vision transformers
45. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding
46. CvT: Introducing convolutions to vision transformers
47. MViTv2: Improved multiscale vision transformers for classification and detection
48. ViTAE: Vision transformer advanced by exploring intrinsic inductive bias
49. Visformer: The vision-friendly transformer
50. Quadtree attention for vision transformers
51. HR-NAS: Searching efficient high-resolution neural architectures with lightweight transformers
52. Inception transformer
53. MCMAE: Masked convolution meets masked autoencoders
54. X. Li, W. Wang, L. Yang, and J. Yang, “Uniform masking: Enabling mae pre-training for pyramid-based vision transformers with locality,” arXiv preprint arXiv:2205.10063, 2022.
55. Vision transformer adapter for dense predictions
56. VOLO: Vision outlooker for visual recognition
57. U-Net: Convolutional networks for biomedical image segmentation
58. J. Chen et al., “TransUNet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306, 2021.
59. UNETR: Transformers for 3D medical image segmentation
60. U-Net Transformer: Self and cross attention for medical image segmentation
61. UTNet: A hybrid transformer architecture for medical image segmentation
62. Y. Gao, M. Zhou, D. Liu, and D. Metaxas, “A multi-scale transformer for medical image segmentation: Architectures, model efficiency, and benchmarks,” arXiv preprint arXiv:2203.00131, 2022.
63. Mixed transformer U-Net for medical image segmentation
64. UCTransNet: Rethinking the skip connections in U-Net from a channel-wise perspective with transformer
65. TransFuse: Fusing transformers and CNNs for medical image segmentation
66. Swin-Unet: Unet-like pure transformer for medical image segmentation
67. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images
68. A robust volumetric transformer for accurate 3D tumor segmentation
69. Segmenter: Transformer for semantic segmentation
70. TopFormer: Token pyramid transformer for mobile semantic segmentation
71. MobileNetV2: Inverted residuals and linear bottlenecks
72. Multi-scale high-resolution vision transformer for semantic segmentation
73. H. Yan, C. Zhang, and M. Wu, “Lawin transformer: Improving semantic segmentation transformer with multi-scale representations via large window attention,” arXiv preprint arXiv:2201.01615, 2022.
74. G. Sharir, A. Noy, and L. Zelnik-Manor, “An image is worth 16 × 16 words, what is a video worth?,” arXiv preprint arXiv:2103.13915, 2021.
75. Frozen in time: A joint video and image encoder for end-to-end retrieval
76. Is space-time attention all you need for video understanding?
77. Token shift transformer for video classification
78. VidTr: Video transformer without convolutions
79. ViViT: A video vision transformer
80. Space-time mixing attention for video transformer
81. TSM: Temporal shift module for efficient video understanding
82. Z. Liu et al., “ConvTransformer: A convolutional transformer network for video frame synthesis,” arXiv preprint arXiv:2011.10185, 2011.
83. End-to-End video instance segmentation with transformers
84. Temporally efficient vision transformer for video instance segmentation
85. Video instance segmentation using inter-frame communication transformers
86. Multiview transformers for video recognition
87. Video transformer network
88. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2004.
89. Anticipative video transformer
90. Multiscale vision transformers
91. Event-based video reconstruction using transformer
92. Cross-frame transformer-based spatiotemporal video super-resolution10.1109/TBC.2022.3147145 ↗
93. RSTT: Real-time spatial temporal transformer for space-time video super-resolution
94. R. Liu et al., “Decoupled spatial-temporal transformer for video inpainting,” arXiv preprint arXiv:2104.06637, 2021.
95. VDTR: Video deblurring with transformer10.1109/TCSVT.2022.3201045 ↗
96. Denoising diffusion probabilistic models
97. Scalable diffusion models with transformers
98. Swinv2-Imagen: Hierarchical vision transformer diffusion models for text-to-image generation
99. One transformer fits all distributions in multi-modal diffusion at scale
100. ET-DM: Text to image via diffusion model with efficient Transformer10.1016/j.displa.2023.102568 ↗