Deep Learning Methods in Image Matting: A Survey
Source abstract
Abstract
Image matting is a fundamental technique used to extract a fine foreground image from a given image by estimating the opacity values of each pixel. It is one of the key techniques in image processing and has a wide range of applications in practical scenarios, such as in image and video editing. Deep learning has demonstrated outstanding performance in various image processing tasks, making it a popular research topic. In recent years, image matting methods based on deep learning have gained significant attention due to their superior performance. Therefore, this article presents a comprehensive overview of the deep learning-based image matting algorithms that have been proposed in recent years. This paper initially introduces frequently used datasets and their production methods, along with the basic principles of traditional image matting techniques. We then analyze deep learning-based matting algorithms in detail and introduce commonly used image matting evaluation metrics. Additionally, this paper discusses the application scenarios of image matting, conducts experiments to illustrate the limitations of current image matting methods, and outlines potential future research directions in this field. Overall, this paper can serve as a valuable reference for researchers that are interested in image matting.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Enhancement Techniques · Image and Signal Denoising Methods · Advanced Image Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin 541004, China
Funding
Guangxi Science and Technology Major Project · Youth Science Fund Project of Guangxi Natural Science Foundation
References (75)
Show source-provided references
1. Smith, A.R., and Blinn, J.F. (1996, January 4–9). Blue screen matting. Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA.10.1145/237170.237263 ↗
2. Mishima, Y. (1994). Soft Edge Chroma-Key Generation Based Upon Hexoctahedral Color Space. (5,355,174), US Patent.
3. Sun, J., Jia, J., Tang, C.K., and Shum, H.Y. (2004). ACM SIGGRAPH 2004 Papers, ACM.
4. A survey on natural image matting with closed-form solutions10.1109/ACCESS.2019.2941832 ↗
5. Boda, J., and Pandya, D. (2018, January 3–5). A survey on image matting techniques. Proceedings of the 2018 International Conference on Communication and Signal Processing (ICCSP), Chennai, India.10.1109/ICCSP.2018.8523834 ↗
6. A survey on pre-processing in image matting10.1007/s11390-017-1709-z ↗
7. Image and video matting: A survey10.1561/0600000019 ↗
8. Shen, X., Tao, X., Gao, H., Zhou, C., and Jia, J. (2016, January 11–14). Deep automatic portrait matting. Proceedings of the Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_6 ↗
9. Xu, N., Price, B., Cohen, S., and Huang, T. (2017, January 21–26). Deep image matting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.41 ↗
10. Chen, Q., Ge, T., Xu, Y., Zhang, Z., Yang, X., and Gai, K. (2018, January 22–26). Semantic human matting. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Republic of Korea.10.1145/3240508.3240610 ↗
11. Active matting
12. Lutz, S., Amplianitis, K., and Smolic, A. (2018). Alphagan: Generative adversarial networks for natural image matting. arXiv.
13. Cai, S., Zhang, X., Fan, H., Huang, H., Liu, J., Liu, J., Liu, J., Wang, J., and Sun, J. (November, January 27). Disentangled image matting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
14. Hou, Q., and Liu, F. (November, January 27). Context-aware image matting for simultaneous foreground and alpha estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
15. Tang, J., Aksoy, Y., Oztireli, C., Gross, M., and Aydin, T.O. (2019, January 15–20). Learning-based sampling for natural image matting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00317 ↗
16. Lu, H., Dai, Y., Shen, C., and Xu, S. (November, January 27). Indices matter: Learning to index for deep image matting. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
17. Tang, H., Huang, Y., Fan, Y., and Zeng, X. (2019, January 22–25). Very deep residual network for image matting. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803682 ↗
18. Zhang, Y., Gong, L., Fan, L., Ren, P., Huang, Q., Bao, H., and Xu, W. (2019, January 15–20). A late fusion cnn for digital matting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00765 ↗
19. Qiao, Y., Liu, Y., Yang, X., Zhou, D., Xu, M., Zhang, Q., and Wei, X. (2020, January 13–19). Attention-guided hierarchical structure aggregation for image matting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01369 ↗
20. Sharma, R., Deora, R., and Vishvakarma, A. (September, January 31). AlphaNet: An attention guided deep network for automatic image matting. Proceedings of the 2020 International Conference on Omni-Layer Intelligent Systems (COINS), Barcelona, Spain.10.1109/COINS49042.2020.9191371 ↗
21. Sengupta, S., Jayaram, V., Curless, B., Seitz, S.M., and Kemelmacher-Shlizerman, I. (2020, January 13–19). Background matting: The world is your green screen. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00236 ↗
22. JMNet: A joint matting network for automatic human matting10.1007/s41095-020-0168-6 ↗
23. Smart scribbles for image matting
24. Li, Y., and Lu, H. (2020, January 7–12). Natural image matting via guided contextual attention. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6809 ↗
25. Sun, Y., Tang, C.K., and Tai, Y.W. (2021, January 20–25). Semantic image matting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01097 ↗
26. Yu, H., Xu, N., Huang, Z., Zhou, Y., and Shi, H. (2021, January 2–9). High-resolution deep image matting. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.
27. Liu, Q., Xie, H., Zhang, S., Zhong, B., and Ji, R. (2021, January 20–24). Long-range feature propagating for natural image matting. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475203 ↗
28. Lin, S., Ryabtsev, A., Sengupta, S., Curless, B.L., Seitz, S.M., and Kemelmacher-Shlizerman, I. (2021, January 20–25). Real-time high-resolution background matting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00865 ↗
29. Li, J., Zhang, J., and Tao, D. (2021). Deep automatic natural image matting. arXiv.10.24963/ijcai.2021/111 ↗
30. Li, J., Ma, S., Zhang, J., and Tao, D. (2021, January 20–24). Privacy-preserving portrait matting. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475512 ↗
31. Deora, R., Sharma, R., and Raj, D.S.S. (2021). Salient image matting. arXiv.
32. Wei, T., Chen, D., Zhou, W., Liao, J., Zhao, H., Zhang, W., and Yu, N. (2021, January 20–25). Improved image matting via real-time user clicks and uncertainty estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01512 ↗
33. Modnet: Real-time trimap-free portrait matting via objective decomposition
34. Lin, S., Yang, L., Saleemi, I., and Sengupta, S. (2022, January 3–8). Robust high-resolution video matting with temporal guidance. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV51458.2022.00319 ↗
35. Chen, G., Liu, Y., Wang, J., Peng, J., Hao, Y., Chu, L., Tang, S., Wu, Z., Chen, Z., and Yu, Z. (2022). Pp-matting: High-accuracy natural image matting. arXiv.
36. Bridging composite and real: Towards end-to-end deep image matting10.1007/s11263-021-01541-0 ↗
37. Deep interactive image matting with feature propagation10.1109/TIP.2022.3155958 ↗
38. User-Guided Deep Human Image Matting Using Arbitrary Trimaps10.1109/TIP.2022.3150295 ↗
39. Park, G., Son, S., Yoo, J., Kim, S., and Kwak, N. (2022, January 18–24). Matteformer: Transformer-based image matting via prior-tokens. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01140 ↗
40. Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., and Rott, P. (2009, January 20–25). A perceptually motivated online benchmark for image matting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206503 ↗
41. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
42. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
43. Adobe Inc. (2019, March 06). Available online: https://www.adobe.com/products/photoshop.html.
44. Chuang, Y.Y., Curless, B., Salesin, D.H., and Szeliski, R. (2001, January 8–14). A bayesian approach to digital matting. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA.
45. Wang, J., and Cohen, M.F. (2007, January 17–22). Optimized color sampling for robust matting. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383006 ↗
46. He, K., Rhemann, C., Rother, C., Tang, X., and Sun, J. (2011, January 20–25). A global sampling method for alpha matting. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995495 ↗
47. Shared sampling for real-time alpha matting10.1111/j.1467-8659.2009.01627.x ↗
48. Shahrian, E., and Rajan, D. (2012, January 16–21). Weighted color and texture sample selection for image matting. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247741 ↗
49. Shahrian, E., Rajan, D., Price, B., and Cohen, S. (2013, January 23–28). Improving image matting using comprehensive sampling sets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.88 ↗
50. Random walks for image segmentation10.1109/TPAMI.2006.233 ↗
51. A closed-form solution to natural image matting10.1109/TPAMI.2007.1177 ↗
52. Zheng, Y., and Kambhamettu, C. (October, January 29). Learning based digital matting. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.
53. Spectral matting10.1109/TPAMI.2008.168 ↗
54. Fast approximate nearest neighbors with automatic algorithm configuration
55. Chen, X., Zou, D., Zhou, Z., Zhao, Q., and Tan, P. (2013, January 23–28). Image matting with local and nonlocal smooth priors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.248 ↗
56. KNN matting10.1109/TPAMI.2013.18 ↗
57. Lee, P., and Wu, Y. (2011, January 20–25). Nonlocal matting. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995665 ↗
58. Aksoy, Y., Aydin, T.O., and Pollefeys, M. (2017, January 21–26). Designing effective inter-pixel information flow for natural image matting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.32 ↗
59. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
60. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
61. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
62. U2-Net: Going deeper with nested U-structure for salient object detection10.1016/j.patcog.2020.107404 ↗
63. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
64. Siam, M., Valipour, S., Jagersand, M., and Ray, N. (2017, January 17–20). Convolutional gated recurrent networks for video segmentation. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ITSC.2017.8317600 ↗
65. Wu, H., Zheng, S., Zhang, J., and Huang, K. (2018, January 18–23). Fast end-to-end trainable guided filter. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00197 ↗
66. Niu, L., Cong, W., Liu, L., Hong, Y., Zhang, B., Liang, J., and Zhang, L. (2021). Making images real again: A comprehensive survey on deep image composition. arXiv.
67. Zhang, H., Zhang, J., Perazzi, F., Lin, Z., and Patel, V.M. (2021, January 5–9). Deep image compositing. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00041 ↗
68. Zhao, Y., Price, B., Cohen, S., and Gurari, D. (2019, January 7–11). Guided image inpainting: Replacing an image region by pulling content from another image. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.10.1109/WACV.2019.00166 ↗
69. Medical matting: Medical image segmentation with uncertainty from the matting perspective10.1016/j.compbiomed.2023.106714 ↗
70. Li, X., Yang, T., Hu, Y., Xu, M., Zhang, W., and Li, F. (2017, January 13–16). Automatic tongue image matting for remote medical diagnosis. Proceedings of the 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas City, MO, USA.10.1109/BIBM.2017.8217710 ↗
71. AWM: Adaptive Weight Matting for medical image segmentation
72. Deep matting for cloud detection in remote sensing images10.1109/TGRS.2020.2988265 ↗
73. Ma, D., Wu, R., Xiao, D., and Sui, B. (2023). Cloud Removal from Satellite Images Using a Deep Learning Model with the Cloud-Matting Method. Remote Sens., 15.10.3390/rs15040904 ↗
74. Matting-based automatic and accurate cloud detection for multisource satellite images10.1117/1.JRS.14.026519 ↗
75. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗