Contextual Coefficients Excitation Feature: Focal Visual Representation for Relationship Detection
Source abstract
Abstract
Visual relationship detection (VRD), a challenging task in the image understanding, suffers from vague connection between relationship patterns and visual appearance. This issue is caused by the high diversity of relationship-independent visual appearance, where inexplicit and redundant cues may not contribute to the relationship detection, even confuse the detector. Previous relationship detection models have shown remarkable progress in leveraging external textual information or scene-level interaction to complement relationship detection cues. In this work, we propose Contextual Coefficients Excitation Feature (CCEF), a focal visual representation, which is adaptively recalibrated from original visual feature responses by explicitly modeling the interdependencies between features and their contextual coefficients. Specifically, contextual coefficients are obtained by calculation of both the spatial coefficients and generated-label ones. In addition, a conditional Wasserstein Generative Adversarial Network (WGAN) regularized with a relationship classification loss is designed to alleviate inadequate training of generated-label coefficients due to long tail distribution of relationship. Experimental results demonstrate the effective improvements of our method on relationship detection. In particular, our method improves the recall from 8.5% to 23.2% of predicting unseen relationship from zero-shot set.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China
Funding
National Natural Science Foundation of China
References (45)
Show source-provided references
1. Zhou, P., Ni, B., Geng, C., Hu, J., and Xu, Y. (2018, January 18–22). Scale-transferrable object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00062 ↗
2. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2017). ImageNet Classification with Deep Convolutional Neural Networks, Communications of the ACM.10.1145/3065386 ↗
3. Wang, H., Wang, Q., Gao, M., Li, P., and Zuo, W. (2018, January 18–22). Multi-scale location-aware kernel representation for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00136 ↗
4. Liu, Y., Wang, R., Shan, S., and Chen, X. (2018, January 18–22). Structure inference net: Object detection using scene-level context and instance-level relationships. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00730 ↗
5. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Li, F.-F. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
6. Zhou, G., Zhang, M., Ji, D.H., and Zhu, Q. (2007, January 28–30). Tree kernel-based relation extraction with context-sensitive structured parse tree information. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), Prague, Czech Republic.
7. Zhuang, B., Liu, L., Shen, C., and Reid, I. (2017, January 22–29). Towards context-aware interaction recognition for visual relationship detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.71 ↗
8. Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L. (2016). Visual relationship detection with language priors. European Conference on Computer Vision, Springer.10.1007/978-3-319-46448-0_51 ↗
9. Yu, R., Li, A., Morariu, V.I., and Davis, L.S. (2017, January 22–29). Visual relationship detection with internal and external linguistic knowledge distillation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.121 ↗
10. Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. (2018, January 18–22). Neural motifs: Scene graph parsing with global context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00611 ↗
11. Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein gan. arXiv.
12. Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv.
13. Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., and Gould, S. (2016, January 27–30). Dynamic image networks for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.331 ↗
14. Sadeghi, M.A., and Farhadi, A. (2011, January 20–25). Recognition using visual phrases. Proceedings of the CVPR 2011, Providence, RI, USA.10.1109/CVPR.2011.5995711 ↗
15. Zhu, Y., Jiang, S., and Li, X. (2017, January 10–14). Visual relationship detection with object spatial distribution. Proceedings of the 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019448 ↗
16. Zhang, H., Kyaw, Z., Chang, S.F., and Chua, T.S. (2017, January 21–26). Visual translation embedding network for visual relation detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.331 ↗
17. Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., and Change Loy, C. (2018, January 8–14). Zoom-net: Mining deep feature interactions for visual relationship recognition. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_20 ↗
18. Yang, J., Lu, J., Lee, S., Batra, D., and Parikh, D. (2018, January 8–14). Graph r-cnn for scene graph generation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_41 ↗
19. Li, Y., Ouyang, W., Zhou, B., Shi, J., Zhang, C., and Wang, X. (2018, January 8–14). Factorizable net: An efficient subgraph-based framework for scene graph generation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01246-5_21 ↗
20. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, MIT Press.
21. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A.C. (2017). Improved training of wasserstein gans. Advances in Neural Information Processing Systems, Curran Associates Inc.
22. Kossaifi, J., Tran, L., Panagakis, Y., and Pantic, M. (2018, January 18–22). Gagan: Geometry-aware generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00098 ↗
23. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2018, January 18–22). Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00577 ↗
24. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv.
25. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., and Choo, J. (2018, January 18–22). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00916 ↗
26. Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale gan training for high fidelity natural image synthesis. arXiv.
27. CM-GANs: Cross-modal generative adversarial networks for common representation learning10.1145/3284750 ↗
28. Wang, J., Li, X., and Yang, J. (2018, January 18–22). Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00192 ↗
29. Zhang, G., Kan, M., Shan, S., and Chen, X. (2018, January 8–14). Generative adversarial network with spatial attention for face attribute editing. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_26 ↗
30. Zhao, B., Chang, B., Jie, Z., and Sigal, L. (2018, January 8–14). Modular generative adversarial networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_10 ↗
31. Bucher, M., Herbin, S., and Jurie, F. (2017, January 22–29). Generating visual representations for zero-shot classification. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.
32. Felix, R., Kumar VB, G., Reid, I., and Carneiro, G. (2018, January 8–14). Multi-modal cycle-consistent generalized zero-shot learning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_2 ↗
33. Xian, Y., Lorenz, T., Schiele, B., and Akata, Z. (2018, January 18–22). Feature generating networks for zero-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00581 ↗
34. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
35. Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems, Curran Associates Inc.
36. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
37. Yang, X., Zhang, H., and Cai, J. (2018, January 8–14). Shuffle-then-assemble: Learning object-agnostic visual relationship features. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01258-8_3 ↗
38. Han, C., Shen, F., Liu, L., Yang, Y., and Shen, H.T. (2018, January 22–26). Visual spatial attention network for relationship detection. Proceedings of the 2018 ACM Multimedia Conference on Multimedia Conference, Seoul, Korea.10.1145/3240508.3240611 ↗
39. Jae Hwang, S., Ravi, S.N., Tao, Z., Kim, H.J., Collins, M.D., and Singh, V. (2018, January 18–22). Tensorize, factorize and regularize: Robust visual relationship learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00112 ↗
40. Peyre, J., Sivic, J., Laptev, I., and Schmid, C. (2017, January 22–29). Weakly-supervised learning of visual relations. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.554 ↗
41. Berthelot, D., Schumm, T., and Metz, L. (2017). Began: Boundary equilibrium generative adversarial networks. arXiv.
42. Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., and Paul Smolley, S. (2017, January 22–29). Least squares generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.304 ↗
43. Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O. (2018). Are gans created equal? A large-scale study. Advances in Neural Information Processing Systems, Curran Associates Inc.
44. Bao, J., Chen, D., Wen, F., Li, H., and Hua, G. (2017, January 22–29). CVAE-GAN: Fine-grained image generation through asymmetric training. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.299 ↗
45. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗