Deep Pixel-Level Matching via Attention for Video Co-Segmentation
Source abstract
Abstract
In video object co-segmentation, methods based on patch-level matching are widely leveraged to extract the similarity between video frames. However, these methods can easily lead to pixel misclassification because they reduce the precision of pixel localization; thus, the accuracies of the segmentation results of these methods are deducted. To address this problem, we propose a framework based on deep neural networks and equipped with a new attention module, which is designed for pixel-level matching to segment the object across video frames in this paper. In this attention module, the pixel-level matching step is able to compare the feature value of each pixel from one input frame with that of each pixel from another input frame for computing the similarity between two frames. Then a features fusion step is applied to efficiently fuse the feature maps of each frame with the similarity information for generating dense attention features. Finally, an up-sampling step refines the feature maps for obtaining high quality segmentation results by using these dense attention features. The ObMiC and DAVIS 2016 datasets were utilized to train and test our framework. Experimental results show that our framework achieves higher accuracy than those of other video segmentation methods that perform well in common information extraction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Visual Attention and Saliency Detection · Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Technology, Macau University of Science and Technology, Macao 999078, China
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China
Funding
The Science and Technology Development Fund of Macao SAR
References (28)
Show source-provided references
1. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., and Brox, T. (2017, January 21–26). Flownet 2.0: Evolution of optical flow estimation with deep networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.179 ↗
2. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2009, January 20–25). Dual attention network for scene segmentation. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.
3. Lu, X., Wang, W., Ma, C., Shen, J., Shao, L., and Porikli, F. (2019, January 16–20). See more, know more: Unsupervised video object segmentation with co-attention siamese networks. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00374 ↗
4. Chen, D., Chen, H., and Chang, L. (November, January 29). Video object cosegmentation. Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan.
5. Chiu, W.C., and Fritz, M. (2013, January 23–28). Multi-class video co-segmentation with a generative multi-video model. Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition 2013, Portland, OR, USA.10.1109/CVPR.2013.48 ↗
6. Extracting primary objects by video co-segmentation10.1109/TMM.2014.2363936 ↗
7. Rubio, J.C., Serrat, J., and López, A. (2013, January 5–8). Video co-segmentation. Proceedings of the 2nd Asian Conference on Computer Vision, Okinawa, Japan.
8. Video object cosegmentation via subspace clustering and quadratic pseudo-boolean optimization in an MRF framework10.1109/TMM.2014.2306393 ↗
9. Robust video object cosegmentation10.1109/TIP.2015.2438550 ↗
10. Video object discovery and co-segmentation with extremely weak supervision10.1109/TPAMI.2016.2612187 ↗
11. Video co-saliency guided co-segmentation10.1109/TCSVT.2017.2701279 ↗
12. Deep learning10.1038/nature14539 ↗
13. Tsai, Y.H., Zhong, G., and Yang, M.H. (2016, January 8–16). Semantic co-segmentation in videos. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_46 ↗
14. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
15. Object-based multiple foreground video co-segmentation via multi-state selection graph10.1109/TIP.2015.2442915 ↗
16. Long, L., Shelhamer, E., and Darrell, T. (2015, January 11–18). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 International Conference on Computer Vision, Araucano Park, Las Condes, Chile.10.1109/CVPR.2015.7298965 ↗
17. Zhang, C., Lin, F., Yao, R., and Shen, C. (2019, January 16–20). CAN Net: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00536 ↗
18. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
19. Glorot, X., Bordes, A., and Bengio, Y. (2011, January 3–5). Deep sparse rectifier neural networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Palermo, Italy.
20. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
21. Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv.
22. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3–7). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference Multimedia, Orlando, FL, USA.10.1145/2647868.2654889 ↗
23. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., and Sorkine-Hornung, A. (July, January 26). A benchmark dataset and evaluation methodology for video object segmentation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.85 ↗
24. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
25. Lee, Y.J., Kim, J., and Grauman, K. (2011, January 6–13). Key-segments for video object segmentation. Proceedings of the 13th International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126471 ↗
26. Papazoglou, A., and Ferrari, V. (2013, January 23–28). Fast object segmentation in unconstrained video. Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition 2013, Portland, OR, USA.10.1109/ICCV.2013.223 ↗
27. Marki, N., Perazzi, F., Wang, O., and Sorkine-Hornung, A. (July, January 26). Bilateral space video segmentation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.87 ↗
28. Joint video object discovery and segmentation by coupled dynamic Markov networks10.1109/TIP.2018.2859622 ↗