Spatial Front-Back Relationship Recognition Based on Partition Sorting Network
Source abstract
Abstract
The current development in computer vision highlights the significance of comprehending the semantic features of images. However, the spatial front-back relationships between objects, which constitute a fundamental semantic feature, have received limited attention. To address this gap, we propose a novel neural network termed Spatial Front-Back Relationship Partition Sorting Network (SFBR-PSortNet), specifically designed for recognizing spatial front-back relationships among objects within images. SFBR-PSortNet is an end-to-end deep convolutional neural network that systematically generates a set of triples representing the spatial front-back relationships between every pair of objects in an input image. The key technical innovations of SFBR-PSortNet include the utilization of bottom keypoints of objects, which serve a dual purpose of enabling object category recognition and providing implicit depth information to enhance spatial front-back relationship reasoning, and the introduction of a Partition Sorting mechanism to construct a comprehensive spatial front-back relationship graph among all objects. Extensive experiments conducted on data derived from the KITTI dataset demonstrate the effectiveness of our network for spatial front-back relationship recognition, achieving a Precision of 0.876 and a Recall of 0.856, respectively. The results validate the practical applicability and robustness of our network in real-world road scenarios, underscoring its potential to enhance the accuracy of computer vision systems in complex environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Marine Electrical Engineering College, Dalian Maritime University, Dalian 116026, China
Information Science and Technology College, Dalian Maritime University, Dalian 116026, China
Funding
National Natural Science Foundation of China · National Key Research, and Development Program of China · Distinguished Young Scholar Project of Dalian City · Fundamental Research Funds for the Central Universities · Dalian City Science and Technology Plan (Key) Project
References (50)
Show source-provided references
1. Sadeghi, M.A., and Farhadi, A. (2011). Recognition Using Visual Phrases, IEEE.10.1109/CVPR.2011.5995711 ↗
2. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
3. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
4. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
5. Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A., and Bengio, Y. (2015). Renet: A recurrent neural network based alternative to convolutional networks. arXiv.
6. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 22–25). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
7. Tan, M., and Le, Q. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
8. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
9. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
10. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, Faster, Stronger. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
11. Farhadi, A., and Redmon, J. (2018, January 18–22). YOLOv3: An Incremental Improvement. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
12. CornerNet: Detecting Objects as Paired Keypoints10.1007/s11263-019-01204-1 ↗
13. Tan, M., Pang, R., and Le, Q.V. (2020, January 14–19). Efficientdet: Scalable and efficient object detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01079 ↗
14. Visual Relationship Detection for Workplace Safety Applications10.1109/TAI.2023.3266183 ↗
15. Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L. (2016, January 11–14). Visual relationship detection with language priors. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_51 ↗
16. Xu, D., Zhu, Y., Choy, C.B., and Fei-Fei, L. (2017, January 21–26). Scene Graph Generation by Iterative Message Passing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.330 ↗
17. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
18. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T. (2017, January 4–9). A simple neural network module for relational reasoning. Proceedings of the Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
19. Sharifzadeh, S., Baharlou, S.M., Berrendorf, M., Koner, R., and Tresp, V. (2021, January 10–15). Improving Visual Relation Detection using Depth Maps. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412945 ↗
20. Liu, X., Gan, M.G., and He, Y. (2022). Multi-view visual relationship detection with estimated depth map. Appl. Sci., 12.10.3390/app12094674 ↗
21. Adaptive depth-aware visual relationship detection10.1016/j.knosys.2022.108786 ↗
22. Yang, J., Lu, J., Lee, S., Batra, D., and Parikh, D. (2018, January 8–14). Graph r-cnn for scene graph generation. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_41 ↗
23. Hudson, D.A., and Manning, C.D. (2018). Compositional attention networks for machine reasoning. arXiv.
24. Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. (2024, January 16–22). SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01370 ↗
25. Wang, H., Liu, X., Li, Y., and Xing, G. (2024, January 12–14). Depth Estimation Method for Monocular Images Combined with Position Estimation. Proceedings of the 2024 13th International Conference of Information and Communication Technology (ICTech), Xiamen, China.10.1109/ICTech63197.2024.00062 ↗
26. Feng, Z., Yang, L., Jing, L., Wang, H., Tian, Y., and Li, B. (2022). Disentangling object motion and occlusion for unsupervised multi-frame monocular depth. European Conference on Computer Vision, Springer Nature.10.1007/978-3-031-19824-3_14 ↗
27. Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals10.1109/TCSVT.2021.3049869 ↗
28. Liu, Z., Wu, Z., and Tóth, R. (2020, January 14–19). SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.10.1109/CVPRW50498.2020.00506 ↗
29. Dual Graph Attention Networks for Multi-View Visual Manipulation Relationship Detection and Robotic Grasping10.1109/TASE.2025.3555206 ↗
30. Lu, Q., Jing, Y., and Zhao, X. (2023). Bolt Loosening Detection Using Key-Point Detection Enhanced by Synthetic Datasets. Appl. Sci., 13.10.3390/app13032020 ↗
31. Zhou, X., Wang, D., and Krähenbühl, P. (2019). Objects as Points. arXiv.
32. Feng, Y., Lei, Y., Yang, X., Xu, J., Liu, X., Xiao, B., and Xu, Y. (2024, January 14–19). SIMMKD: Simple Mask-Flow Keypoint Detection for Both Typhoon Detection and Typhoon Eye Location. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10448466 ↗
33. Online Stochastic Linear Optimization: One-Shot SGD and Beyond
34. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields10.1109/TPAMI.2019.2929257 ↗
35. Zhou, X., Zhuo, J., and Krähenbühl, P. (2019, January 15–20). Bottom-Up Object Detection by Grouping Extreme and Center Points. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00094 ↗
36. Godard, C., Aodha, O.M., Firman, M., and Brostow, G. (November, January 27). Digging into Self-Supervised Monocular Depth Estimation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00393 ↗
37. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Springer.10.1007/978-3-319-24574-4_28 ↗
38. Luo, H., Li, J., Cai, L., and Wu, M. (2023). STrans-YOLOX: Fusing Swin Transformer and YOLOX for Automatic Pavement Crack Detection. Appl. Sci., 13.10.3390/app13031999 ↗
39. ur Rehman, A., Belhaouari, S.B., Kabir, M.A., and Khan, A. (2023). On the Use of Deep Learning for Video Classification. Appl. Sci., 13.10.3390/app13032007 ↗
40. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 19–24). Masked-attention Mask Transformer for Universal Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
41. Na, Y.-H., and Kim, D.-K. (2025). Deep Learning Strategy for UAV-Based Multi-Class Damage Detection on Railway Bridges Using U-Net with Different Loss Functions. Appl. Sci., 15.10.3390/app15158719 ↗
42. Hwang, D., Kim, J.-J., Moon, S., and Wang, S. (2025). Image Augmentation Approaches for Building Dimension Estimation in Street View Images Using Object Detection and Instance Segmentation Based on Deep Learning. Appl. Sci., 15.10.3390/app15052525 ↗
43. Munir, M.A., Khan, M.H., Khan, S., and Khan, F. (2023, January 17–24). Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01104 ↗
44. Xu, R., Chen, C., Peng, J., Li, C., Huang, Y., Song, F., Yan, Y., and Xiong, Z. (2023, January 17–24). Toward RAW Object Detection: A New Benchmark and a New Model. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01286 ↗
45. Alotaibi, A., Alatawi, H., Binnouh, A., Duwayriat, L., Alhmiedat, T., and Alia, O.M. (2024). Deep Learning-Based Vision Systems for Robot Semantic Navigation: An Experimental Study. Technologies, 12.10.3390/technologies12090157 ↗
46. Kim, J., Park, J., Park, J., Kim, J., Kim, S., and Kim, H.J. (2024, January 17–18). Groupwise query specialization and quality-aware multi-assignment for transformer-based visual relationship detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.02660 ↗
47. Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., and Shan, Y. (2024, January 17–18). Yolo-world: Real-time open-vocabulary object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01599 ↗
48. Kweon, H., and Yoon, K.J. (2024, January 17–18). From sam to cams: Exploring segment anything model for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01844 ↗
49. Zhou, H., Greenwood, D., and Taylor, S. (2021). Self-Supervised Monocular Depth Estimation with Internal Feature Fusion. arXiv.10.5244/C.35.208 ↗
50. Bhat, S.F., Alhashim, I., and Wonka, P. (2021, January 20–25). AdaBins: Depth Estimation Using Adaptive Bins. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.