Portrait Semantic Segmentation Method Based on Dual Modal Information Complementarity
Source abstract
Abstract
Semantic segmentation of human images is a research hotspot in the field of computer vision. At present, the semantic segmentation models based on U-net generally lack the ability to capture the spatial information of images. At the same time, semantic incompatibility exists because the feature maps of encoder and decoder are directly connected in the skip connection stage. In addition, in low light scenes such as at night, it is easy for false segmentation and segmentation accuracy to appear. To solve the above problems, a portrait semantic segmentation method based on dual-modal information complementarity is proposed. The encoder adopts a double branch structure, and uses a SK-ASSP module that can adaptively adjust the convolution weights of different receptor fields to extract features in RGB and gray image modes respectively, and carries out cross-modal information complementarity and feature fusion. A hybrid attention mechanism is used in the jump connection phase to capture both the channel and coordinate context information of the image. Experiments on human matting dataset show that the PA and MIoU coefficients of this algorithm model reach 96.58% and 94.48% respectively, which is better than U-net benchmark model and other mainstream semantic segmentation models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Automation, Guangdong University of Technology, Guangzhou 511400, China
College of Computer, Guangdong University of Technology, Guangzhou 511400, China
Funding
Guangdong Provincial Philosophy and Social Science Planning Project of China
References (34)
Show source-provided references
1. Cai, J.L. (2021). Research on Image Semantic Segmentation Technology Based on Deep Learning, Guangdong University of Technology.
2. Wang, X. (2022). Research on Portrait Segmentation Based on Deep Learning, Northwest A&F University.
3. Jia, D.Y. (2022). Semantic Segmentation of Road Potholes Based on Deep Learning, Ningxia University.
4. Algorithm for Portrait Segmentation Combined with MobileNetv2 and Attention Mechanism
5. Urban vegetation classification based on multi-scale feature perception network for UAV images
6. Olaf, R., Philipp, F., and Thomas, B. (2015, January 5–9). U-net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany.
7. MSU-Net: Multi-Scale U-Net for 2D Medical Image Segmentation10.3389/fgene.2021.639930 ↗
8. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
9. Research on Image Semantic Segmentation Algorithm Based on Improved DeepLabv3+
10. Reza, A., Maryam, A., and Mahmood, F. (2020, January 23–28). Attention Deeplabv3+: Multi-level Context Attention Mechanism for Skin Lesion Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.
11. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018). Attention U-net: Learning Where to Look for the Pancreas. arXiv.
12. SCAttNet: Semantic Segmentation Network with Spatial and Channel Attention Mechanism for High-Resolution Remote Sensing Images10.1109/LGRS.2020.2988294 ↗
13. Amer, A., Lambrou, T., and Ye, X. (2022). MDA-Unet: A Multi-Scale Dilated Attention U-Net for Medical Image Segmentation. Appl. Sci., 12.10.3390/app12073676 ↗
14. Zhao, Y., Zhu, D.L., and Ding, H.H. (2024, January 15). Street Scene Real-Time Semantic Segmentation with Fusion Cross Attention Mechanism [J/OL]. Computer Applications and Software. Available online: https://link.cnki.net/urlid/31.1260.TP.20240112.1654.002.
15. Multimodal medical image segmentation using multi-scale context-aware network10.1016/j.neucom.2021.11.017 ↗
16. Multimodal image semantic segmentation based on attention mechanism10.37188/CJLCD.2022-0309 ↗
17. Self-Supervised Model Adaptation for Multimodal Semantic Segmentation10.1007/s11263-019-01188-y ↗
18. Remote Sensing Image Semantic Segmentation Network Based on Multimodal Feature Fusion
19. RGB-D Dual Modal Information Complementary Semantic Segmentation Network
20. Chen, W.Y. (2022). Pedestrian Detection and Scene Segmentation Based on Multimodal Image Fusion, Central South University.
21. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15–20). Dual attention network for scene segmentation. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
22. Images semantic segmentation method based on fusion edge optimization
23. Yuan, Y., Xie, J., Chen, X., and Wang, J. (2020, January 23–28). SegFix: Model-Agnostic Boundary Refinement for Segmentation. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58610-2_29 ↗
24. Bao, L., Yang, Z., Wang, S., Bai, D., and Lee, J. (2020, January 13–19). Real Image Denoising Based on Multi-Scale Residual Dense Block and Cascaded. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00232 ↗
25. Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation
26. A-LinkNet: Semantic segmentation network based on attention and spatial information fusion10.37188/CJLCD.2022-0046 ↗
27. Bischke, B., Helber, P., Folz, J., Borth, D., and Dengel, A. (2019, January 22–25). Multi-task learning for segmentation of building footprints with deep neural networks. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803050 ↗
28. Guo, M., Liu, H., Xu, Y., and Huang, Y. (2020). Building Extraction Based on U-Net with an Attention Block and Multiple Losses. Remote Sens., 12.10.3390/rs12091400 ↗
29. Improved U-net based on Mixed Loss Function for Liver Medical I-mage Segmentation
30. Chen, Q., and Ge, T. (2018, January 22–26). Semantic human matting. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Republic of Korea.10.1145/3240508.3240610 ↗
31. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). CBAM: Convolutional Block Attention Module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
32. Chaurasia, A., and Culurciello, E. (2017, January 10–13). LinkNet: Exploiting encoder representations for efficient semantic segmentation. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), St. Petersburg, FL, USA.10.1109/VCIP.2017.8305148 ↗
33. PortraitNet: Realtime portrait segmentation network for mobile device10.1016/j.cag.2019.03.007 ↗
34. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021, January 20–25). TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.