Research on Weakly Supervised Face Segmentation Technology Based on Visual Large Models in New Media Post-Production
Source abstract
Abstract
Face segmentation is a critical component in new media post-production, enabling the precise separation of facial regions from complex backgrounds at the pixel level. With the increasing demand for flexible and efficient segmentation solutions across diverse media scenarios—such as variety shows, period dramas, and other productions—there is a pressing need for adaptable methods that can perform reliably under varying conditions. However, existing approaches primarily depend on fully supervised learning, which requires extensive manual annotation and incurs high labor costs. To overcome these limitations, we propose a novel weakly supervised face segmentation framework that leverages large-scale vision models to automatically generate high-quality pseudo-labels. These pseudo-labels are then used to train segmentation networks in a dual-model architecture, where two complementary models collaboratively enhance segmentation performance. Our method significantly reduces the reliance on manual labeling while maintaining competitive accuracy. Extensive experiments demonstrate that our approach not only improves segmentation precision and efficiency but also streamlines post-production workflows, lowering human effort and accelerating project timelines. Furthermore, this framework reduced reliance on annotations in the field of weakly supervised learning for facial image processing in the new media post-production scenario.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Visual Attention and Saliency Detection · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China
References (49)
Show source-provided references
1. Du, C., Yu, C., Wang, T., and Zhang, F. (2022). Impact of Virtual Imaging Technology on Film and Television Production Education of College Students Based on Deep Learning and Internet of Things. Front. Psychol., 12.10.3389/fpsyg.2021.766634 ↗
2. The Matter of New Media Making
3. New Media Marketing Strategy Optimization in the Catering Industry Based on Deep Machine Learning Algorithms
4. Chan, K.-H., Pau, G., and Im, S.-K. (2021, January 13–15). Chebyshev Pooling: An Alternative Layer for the Pooling of CNNs-Based Classifier. Proceedings of the 2021 IEEE 4th International Conference on Computer and Communication Engineering Technology (CCET), Beijing, China.10.1109/CCET52649.2021.9544405 ↗
5. Speech emotion recognition based on Graph-LSTM neural network10.1186/s13636-023-00303-9 ↗
6. Levi, G., and Hassner, T. (2015, January 9–13). Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.10.1145/2818346.2830587 ↗
7. Liu, S., Yang, J., Huang, C., and Yang, M.-H. (2015, January 7–12). Multi-objective convolutional learning for face labeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
8. Sung, J., and Chadfield, Z. (2025, June 15). How Technology Can Boost Happiness in Post Production Teams. TVB Europe: Europe’s Television Technology Business Magazine. 2022, 7-7. Available online: https://issuu.com/articles/16368903.
9. Transfiguring portraits
10. Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., and Medioni, G. (2018, January 15–19). On Face Segmentation, Face Swapping, and Face Perception. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi’an, China.10.1109/FG.2018.00024 ↗
11. Nirkin, Y., Keller, Y., and Hassner, T. (November, January 27). FSGAN: Subject Agnostic Face Swapping and Reenactment. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
12. Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., and Samaras, D. (2017, January 21–26). Neural face editing with intrinsic image disentangling. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.578 ↗
13. Mathai, J., Masi, I., and Abdalmageed, W. (2019, January 4–7). Does Generative Face Completion Help Face Recognition?. Proceedings of the 2019 International Conference on Biometrics (ICB), Crete, Greece.10.1109/ICB45273.2019.8987388 ↗
14. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.-Y. (2023, January 1–6). Segment Anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
15. Fully Convolutional Networks for Semantic Segmentation
16. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015, Springer International Publishing.10.1007/978-3-319-24574-4_28 ↗
17. Mask R-CNN10.1109/TPAMI.2018.2844175 ↗
18. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
19. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019). Deep High-Resolution Representation Learning for Human Pose Estimation. arXiv.10.1109/CVPR.2019.00584 ↗
20. Luo, L., Xue, D., Feng, X., Yu, Y., and Wang, P. (2019). Real-time Segmentation and Facial Skin Tones Grading. arXiv.
21. Lee, J., Kim, E., and Yoon, S. (2021, January 20–25). Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00406 ↗
22. Zhang, T., Lin, G., Liu, W., Cai, J., and Kot, A. (2020). Splitting vs. Merging: Mining Object Regions with Discrepancy and Intersection Loss for Weakly Supervised Semantic Segmentation, Springer.10.1007/978-3-030-58542-6_40 ↗
23. Jiang, P.-T., Hou, Q., Cao, Y., Cheng, M.-M., Wei, Y., and Xiong, H. (2019–2, January 27). Integral Object Mining via Online Attention Accumulation. International Conference on Computer Vision. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00216 ↗
24. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
25. Image Segmentation Using Deep Learning: A Survey
26. Lee, J., Yi, J., Shin, C., and Yoon, S. (2021, January 20–25). BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00267 ↗
27. Song, C., Huang, Y., Ouyang, W., and Wang, L. (2019, January 15–20). Box-driven Class-wise Region Masking and Filling Rate Guided Loss for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00325 ↗
28. Sun, W., Zhang, J., and Barnes, N. (December, January 30). 3D Guided Weakly Supervised Semantic Segmentation. Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan.
29. Lin, D., Dai, J., Jia, J., He, K., and Sun, J. (2016, January 27–30). ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.344 ↗
30. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27–30). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.319 ↗
31. Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning10.1007/s11263-020-01293-3 ↗
32. Khoreva, A., Benenson, R., Hosang, J., Hein, M., and Schiele, B. (2017, January 21–26). Simple Does It: Weakly Supervised Instance and Semantic Segmentation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.181 ↗
33. Bearman, A., Russakovsky, O., Ferrari, V., and Fei-Fei, L. What’s the Point: Semantic Segmentation with Point Supervision. Proceedings of the Computer Vision-ECCV 2016, Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016.10.1007/978-3-319-46478-7_34 ↗
34. Zhang, C., Liu, L., Cui, Y., Huang, G., Lin, W., Yang, Y., and Hu, Y. (2023). A Comprehensive Survey on Segment Anything Model for Vision and Beyond. arXiv.
35. Muller, R., Kornblith, S., and Hinton, G. (2019, January 8–14). When Does Label Smoothing Help?. Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, CA, USA.
36. Review of large vision models and visual prompt engineering10.1016/j.metrad.2023.100047 ↗
37. Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping
38. Chen, T., Mai, Z., Li, R., and Chao, W. (2023). Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation. arXiv.
39. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., and Liang, J. (2018, January 20). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. Proceedings of the 4th Deep Learning in Medical Image Analysis (DLMIA) Workshop, Granada, Spain.10.1007/978-3-030-00889-5_1 ↗
40. Laine, S., and Aila, T. (2016). Temporal Ensembling for Semi-Supervised Learning. arXiv.
41. The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale10.1007/s11263-020-01316-z ↗
42. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft COCO: Common Objects in Context, Springer International Publishing.10.1007/978-3-319-10602-1_48 ↗
43. Gupta, A., Dollar, P., and Girshick, R. (2019, January 15–20). LVIS: A Dataset for Large Vocabulary Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00550 ↗
44. LabelMe: A Database and Web-Based Tool for Image Annotation10.1007/s11263-007-0090-8 ↗
45. Weakly Supervised Cell Segmentation by Point Annotation10.1109/TMI.2020.3046292 ↗
46. Zhang, H., Burrows, L., Meng, Y., Sculthorpe, D., Mukherjee, A., E Coupland, S., Chen, K., and Zheng, Y. (2023, January 17–24). Weakly supervised segmentation with point annotations for histopathology images via contrast-based variational model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01500 ↗
47. Kerssies, T., Cavagnero, N., Hermans, A., Norouzi, N., Averta, G., Leibe, B., Dubbelman, G., and de Geus, D. (2025, January 11–15). Your ViT is Secretly an Image Segmentation Model. Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA.
48. Chen, X., Yuan, Y., Zeng, G., and Wang, J. (2021, January 20–25). Semi-supervised semantic segmentation with cross pseudo supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00264 ↗
49. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., and Huang, F. (2023). Qwen technical report. arXiv.