Semantic Matching Based on Semantic Segmentation and Neighborhood Consensus
Source abstract
Abstract
Establishing dense correspondences across semantically similar images is a challenging task, due to the large intra-class variation caused by the unconstrained setting of images, which is prone to cause matching errors. To suppress potential matching ambiguity, NCNet explores the neighborhood consensus pattern in the 4D space of all possible correspondences, which is based on the assumption that the correspondence is continuous in space. We retain the neighborhood consensus constraint, while introducing semantic segmentation information into the features, which makes them more distinguishable and reduces matching ambiguity from a feature perspective. Specifically, we combine the semantic segmentation network to extract semantic features and the 4D convolution to explore 4D-space context consistency. Experiments demonstrate that our algorithm has good semantic matching performances and semantic segmentation information can improve semantic matching accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Medical Image Segmentation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Opto-Electronics Information Technology, Ministry of Education, School of Precision Instrument & Opto-Electronics Engineering, Tianjin University, Tianjin 300072, China
Funding
Tianjin Municipal Transportation Commission Science and Technology Development Plan Project
References (48)
Show source-provided references
1. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms10.1023/A:1014573219977 ↗
2. Liu, H., Wang, R., Xia, Y., and Zhang, X. (2020). Improved Cost Computation and Adaptive Shape Guided Filter for Local Stereo Matching of Low Texture Stereo Images. Appl. Sci., 10.10.3390/app10051869 ↗
3. Crosspatch-based rolling label expansion for dense stereo matching10.1109/ACCESS.2020.2985106 ↗
4. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. (2004, January 11–14). High accuracy optical flow estimation based on a theory for warping. Proceedings of the European Conference on Computer Vision, Prague, Szech Republic.10.1007/978-3-540-24673-2_3 ↗
5. Determining optical flow10.1016/0004-3702(81)90024-2 ↗
6. Long, J.L., Zhang, N., and Darrell, T. (2014, January 8–13). Do convnets learn correspondence?. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
7. Kanazawa, A., Jacobs, D.W., and Chandraker, M. (2016, January 27–30). Warpnet: Weakly supervised matching for single-view reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.354 ↗
8. Choy, C.B., Gwak, J., Savarese, S., and Chandraker, M. (2016). Universal correspondence network. arXiv.
9. Proposal flow: Semantic correspondences from object proposals10.1109/TPAMI.2017.2724510 ↗
10. Jeon, S., Kim, S., Min, D., and Sohn, K. (2018, January 8–14). Parn: Pyramidal affine regression networks for dense semantic correspondence. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_22 ↗
11. Liao, J., Yao, Y., Yuan, L., Hua, G., and Kang, S.B. (2017). Visual attribute transfer through deep image analogy. arXiv.10.1145/3072959.3073683 ↗
12. Sift flow: Dense correspondence across scenes and its applications10.1109/TPAMI.2010.147 ↗
13. Neural best-buddies: Sparse cross-domain correspondence10.1145/3197517.3201332 ↗
14. Deep exemplar-based colorization
15. Bansal, A., Sheikh, Y., and Ramanan, D. (2017). Pixelnn: Example-based image synthesis. arXiv.
16. Zhang, P., Zhang, B., Chen, D., Yuan, L., and Wen, F. (2020, January 13–19). Cross-domain correspondence learning for exemplar-based image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00519 ↗
17. Zhang, Z., Wang, Z., Lin, Z., and Qi, H. (2019, January 15–20). Image super-resolution by neural texture transfer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00817 ↗
18. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
19. Daisy: An efficient dense descriptor applied to wide-baseline stereo10.1109/TPAMI.2009.77 ↗
20. Dalal, N., and Triggs, B. (2005, January 20–26). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
21. Zhou, T., Krahenbuhl, P., Aubry, M., Huang, Q., and Efros, A.A. (2016, January 27–30). Learning dense correspondence via 3d-guided cycle consistency. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.20 ↗
22. Kim, S., Min, D., Ham, B., Jeon, S., Lin, S., and Sohn, K. (2017, January 21–27). Fcss: Fully convolutional self-similarity for dense semantic correspondence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.73 ↗
23. Ham, B., Cho, M., Schmid, C., and Ponce, J. (2016, January 27–30). Proposal flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.378 ↗
24. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
25. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
26. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
27. Schaffalitzky, F., and Zisserman, A. (2002, January 18–19). Automated scene matching in movies. Proceedings of the International Conference on Image and Video Retrieval, London, UK.10.1007/3-540-45479-9_20 ↗
28. Sivic, J., and Zisserman, A. (2003, January 18–20). Video Google: A text retrieval approach to object matching in videos. Proceedings of the IEEE International Conference on Computer Vision, Madison, WI, USA.10.1109/ICCV.2003.1238663 ↗
29. Sattler, T., Leibe, B., and Kobbelt, L. (October, January 29). SCRAMSAC: Improving RANSAC’s efficiency with a spatial consistency filter. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.10.1109/ICCV.2009.5459459 ↗
30. Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., and Cheng, M.M. (2017, January 21–26). Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.302 ↗
31. Rocco, I., Cimpoi, M., Arandjelović, R., Torii, A., Pajdla, T., and Sivic, J. (2018). Neighbourhood consensus networks. arXiv.
32. Novotny, D., Larlus, D., and Vedaldi, A. (2017, January 21–26). Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.306 ↗
33. Huang, S., Wang, Q., Zhang, S., Yan, S., and He, X. (2019, January 27–28). Dynamic context correspondence network for semantic alignment. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.10.1109/ICCV.2019.00210 ↗
34. Lee, J., Kim, D., Ponce, J., and Ham, B. (2019, January 15–20). Sfnet: Learning object-aware semantic correspondence. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00238 ↗
35. Han, K., Rezende, R.S., Ham, B., Wong, K.Y.K., Cho, M., Schmid, C., and Ponce, J. (2017, January 22–29). Scnet: Learning semantic correspondence. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.203 ↗
36. Shechtman, E., and Irani, M. (2007, January 18–23). Matching local self-similarities across images and videos. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383198 ↗
37. Kim, S., Min, D., Lin, S., and Sohn, K. (2016, January 8–16). Deep self-correlation descriptor for dense cross-modal correspondence. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_41 ↗
38. Kim, S., Min, D., Ham, B., Ryu, S., Do, M.N., and Sohn, K. (2015, January 7–12). DASC: Dense adaptive self-correlation descriptor for multi-modal and multi-spectral correspondence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
39. Kim, S., Min, D., Lin, S., and Sohn, K. (2017, January 22–29). Dctm: Discrete-continuous transformation matching for semantic flow. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.485 ↗
40. Rocco, I., Arandjelovic, R., and Sivic, J. (2017, January 21–26). Convolutional neural network architecture for geometric matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.12 ↗
41. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
42. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., and Zisserman, A. (2021, January 10). The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. Available online: http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
43. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2020, December 12). Automatic Differentiation in Pytorch. Available online: http://https://pytorch.org/.
44. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., and Zisserman, A. (2021, January 20). The PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results. Available online: http://www.pascal-network.org/challenges/VOC/voc2011/workshop/index.html.
45. Recurrent transformer networks for semantic correspondence
46. Seo, P.H., Lee, J., Jung, D., Han, B., and Cho, M. (2018, January 8–14). Attentive semantic alignment with offset-aware correlation kernels. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01225-0_22 ↗
47. Rocco, I., Arandjelović, R., and Sivic, J. (2018, January 18–22). End-to-end weakly-supervised semantic alignment. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00723 ↗
48. Jeon, S., Min, D., Kim, S., and Sohn, K. (November, January 27). Joint learning of semantic alignment and object landmark detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.