A Survey on Visual Mamba
Source abstract
Abstract
State space models (SSM) with selection mechanisms and hardware-aware architectures, namely Mamba, have recently shown significant potential in long-sequence modeling. Since the complexity of transformers’ self-attention mechanism is quadratic with image size, as well as increasing computational demands, researchers are currently exploring how to adapt Mamba for computer vision tasks. This paper is the first comprehensive survey that aims to provide an in-depth analysis of Mamba models within the domain of computer vision. It begins by exploring the foundational concepts contributing to Mamba’s success, including the SSM framework, selection mechanisms, and hardware-aware design. Then, we review these vision Mamba models by categorizing them into foundational models and those enhanced with techniques including convolution, recurrence, and attention to improve their sophistication. Furthermore, we investigate the widespread applications of Mamba in vision tasks, which include their use as a backbone in various levels of vision processing. This encompasses general visual tasks, medical visual tasks (e.g., 2D/3D segmentation, classification, image registration, etc.), and remote sensing visual tasks. In particular, we introduce general visual tasks from two levels: high/mid-level vision (e.g., object detection, segmentation, video classification, etc.) and low-level vision (e.g., image super-resolution, image restoration, visual generation, etc.). We hope this endeavor will spark additional interest within the community to address current challenges and further apply Mamba models in computer vision.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Memory and Neural Computing · CCD and CMOS Imaging Sensors · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Automotive Software Innovation Center, Chongqing 401331, China
Institute of Intelligent Software, Guangzhou 511458, China
Department of Computer Science, Saarland University, 66424 Homburg, Germany
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou 310024, China
School of Cyber Space and Technology, University of Science and Technology of China, Hefei 230026, China
Department of Computer Science, University of Oxford, Oxford OX3 7LD, UK
References (100+)
Show source-provided references
1. Rosenblatt, F. (1957). The Perceptron, a Perceiving and Recognizing Automaton Project Para, Cornell Aeronautical Laboratory.
2. Rosenblatt, F., Jones, B., Smith, T., Brown, C., Green, M., Wilson, A., Taylor, J., White, P., King, R., and Johnson, L. (1962). Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, Spartan Books.10.21236/AD0256582 ↗
3. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
4. Imagenet classification with deep convolutional neural networks
5. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
6. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
7. Parikh, A.P., Täckström, O., Das, D., and Uszkoreit, J. (2016). A decomposable attention model for natural language inference. arXiv.10.18653/v1/D16-1244 ↗
8. Attention is all you need
9. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
10. Language models are few-shot learners
11. Gu, A., and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv.
12. Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., and Shalev-Shwartz, S. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv.
13. Pióro, M., Ciebiera, K., Król, K., Ludziejewski, J., and Jaszczur, S. (2024). Moe-mamba: Efficient selective state space models with mixture of experts. arXiv.
14. Anthony, Q., Tokpanov, Y., Glorioso, P., and Millidge, B. (2024). BlackMamba: Mixture of Experts for State-Space Models. arXiv.
15. Fu, D.Y., Dao, T., Saab, K.K., Thomas, A.W., Rudra, A., and Ré, C. (2022). Hungry hungry hippos: Towards language modeling with state space models. arXiv.
16. Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv.
17. Ramachandran, P., Zoph, B., and Le, Q.V. (2017). Swish: A Self-Gated Activation Function. arXiv.
18. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
19. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. (2023). Retentive network: A Successor to Transformer for Large Language Models. arXiv.
20. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020, January 13–18). Transformers are rnns: Fast autoregressive transformers with linear attention. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event.
21. Poli, M., Massaroli, S., Nguyen, E., Fu, D.Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. (2023, January 23–29). Hyena hierarchy: Towards larger convolutional language models. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA.
22. Romero, D.W., Kuzina, A., Bekkers, E.J., Tomczak, J.M., and Hoogendoorn, M. (2021). Ckconv: Continuous kernel convolution for sequential data. arXiv.
23. Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., and Susskind, J. (2021). An attention free transformer. arXiv.
24. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., and GV, K.K. (2023). Rwkv: Reinventing rnns for the transformer era. arXiv.10.18653/v1/2023.findings-emnlp.936 ↗
25. Tallec, C., and Ollivier, Y. (2018). Can recurrent neural networks warp time?. arXiv.
26. Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X. (2024). Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv.
27. Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. (2024). Vmamba: Visual state space model. arXiv.
28. Yang, C., Chen, Z., Espinosa, M., Ericsson, L., Wang, Z., Liu, J., and Crowley, E.J. (2024). PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition. arXiv.
29. Huang, T., Pei, X., You, S., Wang, F., Qian, C., and Xu, C. (2024). LocalMamba: Visual State Space Model with Windowed Selective Scan. arXiv.
30. Pei, X., Huang, T., and Xu, C. (2024). EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba. arXiv.
31. Hu, V.T., Baumann, S.A., Gui, M., Grebenkova, O., Ma, P., Fischer, J., and Ommer, B. (2024). Zigma: Zigzag mamba diffusion model. arXiv.
32. Shi, Y., Xia, B., Jin, X., Wang, X., Zhao, T., Xia, X., Xiao, X., and Yang, W. (2024). VmambaIR: Visual State Space Model for Image Restoration. arXiv.
33. Li, K., Li, X., Wang, Y., He, Y., Wang, Y., Wang, L., and Qiao, Y. (2024). Videomamba: State space model for efficient video understanding. arXiv.
34. Zhang, Z., Liu, A., Reid, I., Hartley, R., Zhuang, B., and Tang, H. (2024). Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv.
35. Yang, Y., Xing, Z., and Zhu, L. (2024). Vivim: A video vision mamba for medical video object segmentation. arXiv.
36. Chen, K., Chen, B., Liu, C., Li, W., Zou, Z., and Shi, Z. (2024). Rsmamba: Remote sensing image classification with state space model. arXiv.10.1109/LGRS.2024.3407111 ↗
37. Behrouz, A., Santacatterina, M., and Zabih, R. (2024). MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection. arXiv.
38. Li, S., Singh, H., and Grover, A. (2024). Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data. arXiv.
39. Patro, B.N., and Agneeswaran, V.S. (2024). SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series. arXiv.
40. Li, S., Zhu, T., Duan, F., Chen, L., Ning, H., and Wan, Y. (2024). HARMamba: Efficient Wearable Sensor Human Activity Recognition Based on Bidirectional Selective SSM. arXiv.
41. Cheng, C., Wang, H., and Sun, H. (2024). Activating Wider Areas in Image Super-Resolution. arXiv.
42. Qiao, Y., Yu, Z., Guo, L., Chen, S., Zhao, Z., Sun, M., Wu, Q., and Liu, J. (2024). VL-Mamba: Exploring State Space Models for Multimodal Learning. arXiv.
43. Chen, G., Huang, Y., Xu, J., Pei, B., Chen, Z., Li, Z., Wang, J., Li, K., Lu, T., and Wang, L. (2024). Video mamba suite: State space model as a versatile alternative for video understanding. arXiv.
44. Liu, J., Yu, R., Wang, Y., Zheng, Y., Deng, T., Ye, W., and Wang, H. (2024). Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv.
45. Wang, J., Chen, J., Chen, D., and Wu, J. (2024). Large Window-based Mamba UNet for Medical Image Segmentation: Beyond Convolution and Self-attention. arXiv.
46. Zhang, Y., Yan, W., Yan, K., Lam, C.P., Qiu, Y., Zheng, P., Tang, R.S.Y., and Cheng, S.S. (2024). Motion-Guided Dual-Camera Tracker for Low-Cost Skill Evaluation of Gastric Endoscopy. arXiv.
47. Tang, Y., Dong, P., Tang, Z., Chu, X., and Liang, J. (2024). VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting. arXiv.10.1109/CVPRW63382.2024.00575 ↗
48. Chen, C.S., Chen, G.Y., Zhou, D., Jiang, D., and Chen, D.S. (2024). Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning. arXiv.
49. Wan, Z., Wang, Y., Yong, S., Zhang, P., Stepputtis, S., Sycara, K., and Xie, Y. (2024). Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation. arXiv.
50. Yang, Y., Ma, C., Yao, J., Zhong, Z., Zhang, Y., and Wang, Y. (2024). ReMamber: Referring Image Segmentation with Mamba Twister. arXiv.
51. Wang, Z., Zheng, J.Q., Zhang, Y., Cui, G., and Li, L. (2024). Mamba-unet: Unet-like pure visual mamba for medical image segmentation. arXiv.
52. Ma, C., and Wang, Z. (2024). Semi-Mamba-UNet: Pixel-Level Contrastive and Pixel-Level Cross-Supervised Visual Mamba-based UNet for Semi-Supervised Medical Image Segmentation. arXiv.10.1016/j.knosys.2024.112203 ↗
53. Wang, Z., Zheng, J.Q., Ma, C., and Guo, T. (2024). VMambaMorph: A Visual Mamba-based Framework with Cross-Scan Module for Deformable 3D Image Registration. arXiv.
54. Chen, H., Song, J., Han, C., Xia, J., and Yokoya, N. (2024). ChangeMamba: Remote Sensing Change Detection with Spatio-Temporal State Space Model. arXiv.10.1109/TGRS.2024.3417253 ↗
55. Wu, R., Liu, Y., Liang, P., and Chang, Q. (2024). H-vmunet: High-order Vision Mamba UNet for Medical Image Segmentation. arXiv.
56. Huang, J., Yang, L., Wang, F., Wu, Y., Nan, Y., Aviles-Rivero, A.I., Schönlieb, C.B., Zhang, D., and Yang, G. (2024). MambaMIR: An Arbitrary-Masked Mamba for Joint Medical Image Reconstruction and Uncertainty Estimation. arXiv.
57. Guo, H., Li, J., Dai, T., Ouyang, Z., Ren, X., and Xia, S.T. (2024). MambaIR: A Simple Baseline for Image Restoration with State-Space Model. arXiv.
58. Shahab Sepehri, M., Fabian, Z., and Soltanolkotabi, M. (2024). Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Models. arXiv.
59. Sanjid, K.S., Hossain, M.T., Junayed, M.S.S., and Uddin, D.M.M. (2024). Integrating Mamba Sequence Model and Hierarchical Upsampling Network for Accurate Semantic Segmentation of Multiple Sclerosis Legion. arXiv.
60. Tang, H., Cheng, L., Huang, G., Tan, Z., Lu, J., and Wu, K. (2024). Rotate to Scan: UNet-like Mamba with Triplet SSM Module for Medical Image Segmentation. arXiv.
61. Liu, J., Yang, H., Zhou, H.Y., Xi, Y., Yu, L., Yu, Y., Liang, Y., Shi, G., Zhang, S., and Zheng, H. (2024). Swin-umamba: Mamba-based unet with imagenet-based pretraining. arXiv.10.1007/978-3-031-72114-4_59 ↗
62. Wu, R., Liu, Y., Liang, P., and Chang, Q. (2024). UltraLight VM-UNet: Parallel Vision Mamba Significantly Reduces Parameters for Skin Lesion Segmentation. arXiv.
63. Ruan, J., and Xiang, S. (2024). Vm-unet: Vision mamba unet for medical image segmentation. arXiv.
64. Zhang, M., Yu, Y., Gu, L., Lin, T., and Tao, X. (2024). VM-UNET-V2 Rethinking Vision Mamba UNet for Medical Image Segmentation. arXiv.10.1007/978-981-97-5128-0_27 ↗
65. Yue, Y., and Li, Z. (2024). MedMamba: Vision Mamba for Medical Image Classification. arXiv.
66. Chen, T., Tan, Z., Gong, T., Chu, Q., Wu, Y., Liu, B., Ye, J., and Yu, N. (2024). MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection. arXiv.10.1109/TGRS.2024.3485721 ↗
67. Ma, X., Zhang, X., and Pun, M.O. (2024). RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation. arXiv.10.1109/LGRS.2024.3414293 ↗
68. Zhen, Z., Hu, Y., and Feng, Z. (2024). FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining. arXiv.
69. Zhao, S., Chen, H., Zhang, X., Xiao, P., Bai, L., and Ouyang, W. (2024). RS-Mamba for Large Remote Sensing Image Dense Prediction. arXiv.10.1109/TGRS.2024.3425540 ↗
70. Gong, H., Kang, L., Wang, Y., Wan, X., and Li, H. (2024). nnmamba: 3d biomedical image segmentation, classification and landmark detection with state space model. arXiv.
71. Xing, Z., Ye, T., Yang, Y., Liu, G., and Zhu, L. (2024). Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv.10.1109/JBHI.2024.3360239 ↗
72. Yang, J.X., Zhou, J., Wang, J., Tian, H., and Liew, A.W.C. (2024). Hsimamba: Hyperpsectral imaging efficient feature learning with bidirectional state space for classification. arXiv.
73. He, X., Cao, K., Yan, K., Li, R., Xie, C., Zhang, J., and Zhou, M. (2024). Pan-Mamba: Effective pan-sharpening with State Space Model. arXiv.
74. Xu, Z., Lin, Y., Han, H., Yang, S., Li, R., Zhang, Y., and Li, X. (2024). MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Models. arXiv.
75. Zhu, Q., Cai, Y., Fang, Y., Yang, Y., Chen, C., Fan, L., and Nguyen, A. (2024). Samba: Semantic Segmentation of Remotely Sensed Images with State Space Model. arXiv.10.1016/j.heliyon.2024.e38495 ↗
76. Wang, Z., and Ma, C. (2024). Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentation. arXiv.
77. Hao, J., He, L., and Hung, K.F. (2024). T-Mamba: Frequency-Enhanced Gated Long-Range Dependency for Tooth 3D CBCT Segmentation. arXiv.
78. Xie, J., Liao, R., Zhang, Z., Yi, S., Zhu, Y., and Luo, G. (2024). ProMamba: Prompt-Mamba for polyp segmentation. arXiv.
79. Zubić, N., Gehrig, M., and Scaramuzza, D. (2024). State Space Models for Event Cameras. arXiv.10.1109/CVPR52733.2024.00556 ↗
80. Islam, M.M., and Bertasius, G. (2022, January 23–28). Long movie clip classification with state-space video models. Proceedings of the European Conference on Computer Vision. Springer, Glasgow, UK.10.1007/978-3-031-19833-5_6 ↗
81. Zheng, Z., and Zhang, J. (2024). FD-Vision Mamba for Endoscopic Exposure Correction. arXiv.
82. Yang, G., Du, K., Yang, Z., Du, Y., Zheng, Y., and Wang, S. (2024). CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification. arXiv.
83. Li, Y., Yang, W., and Fei, B. (2024). 3DMambaComplete: Exploring Structured State Space Model for Point Cloud Completion. arXiv.
84. Ma, J., Li, F., and Wang, B. (2024). U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv.
85. Zheng, Z., and Wu, C. (2024). U-shaped Vision Mamba for Single Image Dehazing. arXiv.
86. Liao, W., Zhu, Y., Wang, X., Pan, C., Wang, Y., and Ma, L. (2024). Lightm-unet: Mamba assists in lightweight unet for medical image segmentation. arXiv.
87. Guo, T., Wang, Y., and Meng, C. (2024). Mambamorph: A mamba-based backbone with contrastive feature learning for deformable mr-ct registration. arXiv.
88. Fei, Z., Fan, M., Yu, C., and Huang, J. (2024). Scalable Diffusion Models with State Space Backbone. arXiv.
89. Oshima, Y., Taniguchi, S., Suzuki, M., and Matsuo, Y. (2024). SSM Meets Video Diffusion Models: Efficient Video Generation with Structured State Spaces. arXiv.
90. Fu, L., Li, X., Cai, X., Wang, Y., Wang, X., Shen, Y., and Yao, Y. (2024). MD-Dose: A Diffusion Model based on the Mamba for Radiotherapy Dose Prediction. arXiv.
91. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
92. Convolutional LSTM network: A machine learning approach for precipitation nowcasting
93. Li, W., Hong, X., and Fan, X. (2024). SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding. arXiv.
94. Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., and Dollár, P. (2020, January 13–19). Designing network design spaces. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01044 ↗
95. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
96. Zhang, P., Dai, X., Yang, J., Xiao, B., Yuan, L., Zhang, L., and Gao, J. (2021, January 10–17). Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00299 ↗
97. Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., and Gao, J. (2021). Focal self-attention for local-global interactions in vision transformers. arXiv.
98. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
99. He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., and Li, M. (2019, January 15–20). Bag of tricks for image classification with convolutional neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00065 ↗
100. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 21–26). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗