Multi-View Visual Relationship Detection with Estimated Depth Map
Source abstract
Abstract
The abundant visual information contained in multi-view images is widely used in computer vision tasks. Existing visual relationship detection frameworks have extended the feature vector to improve model performance. However, single view information can not fully reveal the visual relationships in complex visual scenes. To solve this problem and explore the multi-view information in a visual relationship detection (VRD) model, a novel multi-view VRD framework based on a monocular RGB image and an estimated depth map is proposed. The contributions of this paper are threefold. First, we construct a novel multi-view framework which fuses information of different views extracted from estimated RGB-D images. Second, a multi-view image generation method is proposed to transfer flat visual space to 3D multi-view space. Third, we redesign the visual relationship balanced classifier which can process multi-view feature vectors simultaneously. Detailed experiments were conducted on two datasets to demonstrate the effectiveness of the multi-view VRD framework. The experimental results showed that the multi-view VRD framework resulted in state-of-the-art zero-shot learning performance in specific depth conditions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image and Video Retrieval Techniques · Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Intelligent Control and Decision of Complex Systems, School of Automation, Beijing Institute of Technology, Beijing 100081, China
Funding
National Key R&D Program of China
References (38)
Show source-provided references
1. Visual Relationship Detection with Language Priors
2. Zhang, H., Kyaw, Z., Chang, S., and Chua, T. (2017, January 21–26). Visual Translation Embedding Network for Visual Relation Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, IEEE Computer Society, Honolulu, HI, USA.10.1109/CVPR.2017.331 ↗
3. Yu, R., Li, A., Morariu, V.I., and Davis, L.S. (2017, January 22–29). Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, IEEE Computer Society, Venice, Italy.10.1109/ICCV.2017.121 ↗
4. Peyre, J., Laptev, I., Schmid, C., and Sivic, J. (2017, January 22–29). Weakly-Supervised Learning of Visual Relations. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, IEEE Computer Society, Venice, Italy.10.1109/ICCV.2017.554 ↗
5. Zhang, H., Kyaw, Z., Yu, J., and Chang, S. (2017, January 22–29). PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, IEEE Computer Society, Venice, Italy.10.1109/ICCV.2017.454 ↗
6. Liang, X., Lee, L., and Xing, E.P. (2017, January 21–26). Deep Variation-Structured Reinforcement Learning for Visual Relationship and Attribute Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, IEEE Computer Society, Honolulu, HI, USA.10.1109/CVPR.2017.469 ↗
7. RDBN: Visual relationship detection with inaccurate RGB-D images10.1016/j.knosys.2020.106142 ↗
8. Sharifzadeh, S., Baharlou, S.M., Berrendorf, M., Koner, R., and Tresp, V. (2021, January 10–15). Improving Visual Relation Detection using Depth Maps. Proceedings of the 25th International Conference on Pattern Recognition, ICPR 2020, Milan, Italy.10.1109/ICPR48806.2021.9412945 ↗
9. Thomas, A., Ferrari, V., Leibe, B., Tuytelaars, T., Schiele, B., and Gool, L.V. (2006, January 17–22). Towards Multi-View Object Class Detection. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), New York, NY, USA.
10. Multi-View Saliency Guided Deep Neural Network for 3-D Object Retrieval and Classification10.1109/TMM.2019.2943740 ↗
11. 3D Object Localisation from Multi-View Image Detections
12. Tang, C., Ling, Y., Yang, X., Jin, W., and Zheng, C. (2018). Multi-view object detection based on deep learning. Appl. Sci., 8.10.3390/app8091423 ↗
13. Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21–26). Multi-view 3D Object Detection Network for Autonomous Driving. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, IEEE Computer Society, Honolulu, HI, USA.10.1109/CVPR.2017.691 ↗
14. Knowledge Base Relation Detection via Multi-View Matching
15. Multi-view multitask learning for knowledge base relation detection10.1016/j.knosys.2019.104870 ↗
16. Text Co-Detection in Multi-View Scene10.1109/TIP.2020.2973511 ↗
17. Delaunay triangulation based text detection from multi-view images of natural scene10.1016/j.patrec.2019.11.021 ↗
18. Eigen, D., and Fergus, R. (2015, January 7–13). Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture. Proceedings of the 2015 IEEE International Conference on Computer Vision, ICCV 2015, IEEE Computer Society, Santiago, Chile.10.1109/ICCV.2015.304 ↗
19. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, Lake Tahoe, NV, USA.
20. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
21. Girshick, R.B., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, IEEE Computer Society, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
22. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
23. Plummer, B.A., Mallya, A., Cervantes, C.M., Hockenmaier, J., and Lazebnik, S. (2017, January 22–29). Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, IEEE Computer Society, Venice, Italy.10.1109/ICCV.2017.213 ↗
24. Sadeghi, F., Divvala, S.K., and Farhadi, A. (2015, January 7–12). VisKE: Visual knowledge extraction and question answering by visual verification of relation phrases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, IEEE Computer Society, Boston, MA, USA.10.1109/CVPR.2015.7298752 ↗
25. Qiu, Y., Satoh, Y., Suzuki, R., Iwata, K., and Kataoka, H. (2020). Multi-View Visual Question Answering with Active Viewpoint Selection. Sensors, 20.10.3390/s20082281 ↗
26. Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., and Darrell, T. (2016, January 27–30). Natural Language Object Retrieval. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, IEEE Computer Society, Las Vegas, NV, USA.10.1109/CVPR.2016.493 ↗
27. Johnson, J., Karpathy, A., and Li, F.-F. (2016, January 27–30). DenseCap: Fully Convolutional Localization Networks for Dense Captioning. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.494 ↗
28. Johnson, J., Krishna, R., Stark, M., Li, L., Shamma, D.A., Bernstein, M.S., and Fei-Fei, L. (2015, January 7–12). Image retrieval using scene graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, IEEE Computer Society, Boston, MA, USA.10.1109/CVPR.2015.7298990 ↗
29. Girshick, R.B. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision, ICCV 2015, IEEE Computer Society, Santiago, Chile.10.1109/ICCV.2015.169 ↗
30. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations10.1007/s11263-016-0981-7 ↗
31. Transductive Multi-View Zero-Shot Learning10.1109/TPAMI.2015.2408354 ↗
32. Gu, X., Lin, T., Kuo, W., and Cui, Y. (2021). Zero-Shot Detection via Vision and Language Knowledge Distillation. arXiv.
33. Joint Visual and Semantic Optimization for zero-shot learning10.1016/j.knosys.2021.106773 ↗
34. Hwang, S.J., Ravi, S.N., Tao, Z., Kim, H.J., Collins, M.D., and Singh, V. (2018, January 18–22). Tensorize, Factorize and Regularize: Robust Visual Relationship Learning. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Computer Vision Foundation/IEEE Computer Society, Salt Lake City, UT, USA.10.1109/CVPR.2018.00112 ↗
35. Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
36. Bin, Y., Yang, Y., Tao, C., Huang, Z., Li, J., and Shen, H.T. (February, January 27). MR-NET: Exploiting Mutual Relation for Visual Relationship Detection. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, HI, USA.10.1609/aaai.v33i01.33018110 ↗
37. Zhang, J., Shih, K.J., Elgammal, A., Tao, A., and Catanzaro, B. (2019, January 16–20). Graphical Contrastive Losses for Scene Graph Parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Computer Vision Foundation/IEEE, Long Beach, CA, USA.10.1109/CVPR.2019.01180 ↗
38. Zhan, Y., Yu, J., Yu, T., and Tao, D. (2019, January 16–20). On Exploring Undetermined Relationships for Visual Relationship Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Computer Vision Foundation/IEEE, Long Beach, CA, USA.10.1109/CVPR.2019.00527 ↗