Multi-Head Structural Attention-Based Vision Transformer with Sequential Views for 3D Object Recognition
Source abstract
Abstract
Multi-view image classification tasks require the effective extraction of both spatial and temporal features to fully leverage the complementary information across views. In this study, we propose a lightweight yet powerful model, Multi-head Sparse Structural Attention-based Vision Transformer (MSSAViT), which integrates Structural Self-Attention mechanisms into a compact framework optimized for multi-view inputs. The model employs a fixed MobileNetV3 as a Feature Extraction Module (FEM) to ensure consistent feature patterns across views, followed by Spatial Sparse Self-Attention (SSSA) and Temporal Sparse Self-Attention (TSSA) modules that capture long-range spatial dependencies and inter-view temporal dynamics, respectively. By leveraging these structural attention mechanisms, the model achieves the effective fusion of spatial and temporal information. Importantly, the total model size is reduced to 6.1 M with only 1.5 M trainable parameters, making it highly efficient. Comprehensive experiments demonstrate the proposed model’s superior performance and robustness in multi-view classification tasks, outperforming baseline methods while maintaining a lightweight design. These results highlight the potential of MSSAViT as a practical solution for real-world applications under resource constraints.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Tiandi (Changzhou) Automation Co., Ltd., Changzhou 213015, China
School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China
School of Artificial Intelligence, Beihang University, Beijing 100191, China
Beijing Huahang Institute of Radio Measurement, Beijing 100013, China
Funding
International Science and Technology Cooperation Project
References (49)
Show source-provided references
1. Wang, Q., Song, J., Du, C., and Wang, C. (2024). Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR. Sensors, 24.10.3390/s24144756 ↗
2. Luo, H., Zhang, J., Liu, X., Zhang, L., and Liu, J. (2024). Large-Scale 3D Reconstruction from Multi-View Imagery: A Comprehensive Review. Remote Sens., 16.10.3390/rs16050773 ↗
3. Zhou, L., Tan, J., Fu, J., and Shao, G. (2025). Fast 3D Transmission Tower Detection Based on Virtual Views. Appl. Sci., 15.10.3390/app15020947 ↗
4. Al-Okby, M.F.R., Junginger, S., Roddelkopf, T., and Thurow, K. (2024). UWB-Based Real-Time Indoor Positioning Systems: A Comprehensive Review. Appl. Sci., 14.10.3390/app142311005 ↗
5. Li, J., He, X., Zhou, C., Cheng, X., Wen, Y., and Zhang, D. (October, January 29). ViewFormer: Exploring Spatiotemporal Modeling for Multi-View 3D Occupancy Perception via View-Guided Transformers. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
6. Kim, M., Seo, P.H., Schmid, C., and Cho, M. (2024, January 17–21). Learning Correlation Structures for Vision Transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01792 ↗
7. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M.X., Wang, W.J., Zhu, Y.K., Pang, R.M., and Vasudevan, V. (November, January 27). Searching for mobilenetv3. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
8. Qi, C.R., Su, H., Mo, K.C., and Guibas, L.J. (2017, January 21–26). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.
9. Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017, January 4–9). PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
10. Su, H., Maji, S., Kalogerakis, E., and Learned-Miller, E. (2015, January 17–18). Multi-view Convolutional Neural Networks for 3D Shape Recognition. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.114 ↗
11. Unsupervised 3D local feature learning by circle convolutional restricted Boltzmann machine10.1109/TIP.2016.2605920 ↗
12. Mesh convolutional restricted Boltzmann machines for unsupervised learning of features with structure preservation on 3-D meshes10.1109/TNNLS.2016.2582532 ↗
13. Feng, Y.T., Feng, Y.F., You, H.X., Zhao, X.B., and Gao, Y. (February, January 27). MeshNet: Mesh neural network for 3D shape representation. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Honolulu, HI, USA.
14. MEAN: An attention-based approach for 3D mesh shape classification10.1007/s00371-023-03003-9 ↗
15. Li, J.X., Chen, B.M., and Lee, G.H. (2018, January 18–22). So-net: Self-organizing network for point cloud analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00979 ↗
16. Liu, X.H., Han, Z.Z., Liu, Y.S., and Zwicker, M. (February, January 27). Point2Sequence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Honolulu, HI, USA.
17. Sun, S.F., Rao, Y.M., Lu, J.W., and Yan, H.B. (2024, January 17–21). X-3D: Explicit 3D Structure Modeling for Point Cloud Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00485 ↗
18. Wu, Z.R., Song, S.R., Khosla, A., Yu, F., Zhang, L.G., Tang, X.O., and Xiao, J.X. (2015, January 11–12). 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE International Conference on Computer Vision Workshops (CVPR), Boston, MA, USA.
19. Hamdi, A., Giancola, S., and Ghanem, B. (2021, January 10). Mvtn: Multi-view transformation network for 3d shape recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCV48922.2021.00007 ↗
20. Review of multi-view 3D object recognition methods based on deep learning10.1016/j.displa.2021.102053 ↗
21. Wei, X., Yu, R., and Sun, J. (2020, January 14–19). View-gcn: View-based graph convolutional network for 3d shape analysis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00192 ↗
22. SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by RNN with attention10.1109/TIP.2018.2868426 ↗
23. 3D2SeqViews: Aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation10.1109/TIP.2019.2904460 ↗
24. MVCLN: Multi-View convolutional LSTM network for cross-media 3D shape recognition10.1109/ACCESS.2020.3012692 ↗
25. Alexey, D., Philipp, F., Eddy, I., Philip, H., Caner, H., Vladimir, G., Patrick, V.D.S., Daniel, C., and Thomas, B. (2015, January 13–16). Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.
26. Yang, G.S., and Ramanan, D. (2019, January 8–14). Volumetric Correspondence Networks for Optical Flow. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
27. Human action recognition and prediction: A survey10.1007/s11263-022-01594-9 ↗
28. Human action recognition from various data modalities: A review
29. Anirudh, T., Sanath, N., Salman, K., Rao, M.A., Fahad, S.K., and Bernard, G. (2022, January 19–24). Spatio-temporal relation modeling for few-shot action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.
30. Kim, M.J., Kwon, H., Wang, C.Y., Kwak, S., and Cho, M. (2021, January 6–14). Relational self-attention: What’s missing in attention for video understanding. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
31. Carreira, J., and Zisserman, A. (2017, January 21–27). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
32. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). SlowFast Networks for Video Recognition. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
33. Bertasius, G., Wang, H., and Torresani, L. (2021, January 18–24). Is Space-Time Attention All You Need for Video Understanding?. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
34. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., and Guo, B. (2021, January 10). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCV48922.2021.00986 ↗
35. Yang, Y.Q., Feng, C., Shen, Y.R., and Tian, D. (2018, January 19–21). Foldingnet: Point cloud auto-encoder via deep grid deformation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00029 ↗
36. Maurana, D., and Scherer, S. (October, January 28). Voxnet: A 3d convolutional neural network for real-time object recognition. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7353481 ↗
37. Zhi, S.F., Liu, Y.X., Li, X., and Guo, Y.L. (2017, January 23–24). LightNet: A lightweight 3d convolutional neural network for real-time 3d object recognition. Proceedings of the 10th Eurographics Workshop on 3D Object Retrieval (3DOR), Lyon, France.
38. Sinha, A., Bai, J., and Ramani, K. (2016, January 11–14). Deep learning 3D shape surfaces using geometry images. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46466-4_14 ↗
39. Wang, W., Wang, X., Chen, G., and Zhou, H. (2022). Multi-view SoftPool attention convolutional networks for 3D model classification. Front. Neurorobotics, 16.10.3389/fnbot.2022.1029968 ↗
40. View-relation constrained global representation learning for multi-view-based 3D object recognition10.1007/s10489-022-03949-8 ↗
41. Hegde, V., and Zadeh, R. (2016). Fusionnet: 3d object classification using multiple data representations. arXiv.
42. NormalNet: A voxel-based CNN for 3D object classification and retrieval10.1016/j.neucom.2018.09.075 ↗
43. MVPointNet: Multi-view network for 3D object based on point cloud10.1109/JSEN.2019.2937089 ↗
44. Multi-view dual attention network for 3D object recognition10.1007/s00521-021-06588-1 ↗
45. MVContrast: Unsupervised Pretraining for Multi-view 3D Object Recognition10.1007/s11633-023-1430-z ↗
46. Multi-view convolutional vision transformer for 3D object recognition10.1016/j.jvcir.2023.103906 ↗
47. DILF: Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding10.1016/j.inffus.2023.102033 ↗
48. Alzahrani, M., Usman, M., Anwar, S., and Helmy, T. (2024, January 17–21). Selective Multi-View Deep Model for 3D Object Classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPRW63382.2024.00077 ↗
49. Ren, H., Wang, J., Yang, M., and Velipasalar, S. (2024, January 17–21). PointOfView: A Multi-modal Network for Few-shot 3D Point Cloud Classification Fusing Point and Multi-view Image Features. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPRW63382.2024.00083 ↗