EHANet: An Effective Hierarchical Aggregation Network for Face Parsing
Source abstract
Abstract
In recent years, benefiting from deep convolutional neural networks (DCNNs), face parsing has developed rapidly. However, it still has the following problems: (1) Existing state-of-the-art frameworks usually do not satisfy real-time while pursuing performance; (2) similar appearances cause incorrect pixel label assignments, especially in the boundary; (3) to promote multi-scale prediction, deep features and shallow features are used for fusion without considering the semantic gap between them. To overcome these drawbacks, we propose an effective and efficient hierarchical aggregation network called EHANet for fast and accurate face parsing. More specifically, we first propose a stage contextual attention mechanism (SCAM), which uses higher-level contextual information to re-encode the channel according to its importance. Secondly, a semantic gap compensation block (SGCB) is presented to ensure the effective aggregation of hierarchical information. Thirdly, the advantages of weighted boundary-aware loss effectively make up for the ambiguity of boundary semantics. Without any bells and whistles, combined with a lightweight backbone, we achieve outstanding results on both CelebAMask-HQ (78.19% mIoU) and Helen datasets (90.7% F1-score). Furthermore, our model can achieve 55 FPS on a single GTX 1080Ti card with 640 × 640 input and further reach over 300 FPS with a resolution of 256 × 256, which is suitable for real-world applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Face and Expression Recognition · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China
Funding
National Natural Science Foundation of China
References (39)
Show source-provided references
1. Ou, X., Liu, S., Cao, X., and Ling, H. (2016, January 15–19). Beauty emakeup: A deep makeup transfer system. Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2973812 ↗
2. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B. (2018, January 19–21). High-resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00917 ↗
3. Content-adaptive sketch portrait generation by decompositional representation learning10.1109/TIP.2016.2623485 ↗
4. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
5. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
6. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
7. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke., V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
8. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
9. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
10. Lee, C.H., Liu, Z., Wu, L., and Luo, P. (2019). MaskGAN: Towards diverse and interactive facial image manipulation. arXiv.10.1109/CVPR42600.2020.00559 ↗
11. Smith, B.M., Zhang, L., Brandt, J., Lin, Z., and Yang, J. (2013, January 25–27). Exemplar-based face parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Oregon, Portland.10.1109/CVPR.2013.447 ↗
12. Kae, A., Sohn, K., Lee, H., and Learned-Miller, E. (2013, January 25–27). Augmenting CRFs with Boltzmann machine shape priors for image labeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Oregon, Portland.10.1109/CVPR.2013.263 ↗
13. Liu, S., Yang, J., Huang, C., and Yang, M.H. (2015, January 7–12). Multi-objective convolutional learning for face labeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
14. Liu, S., Shi, J., Liang, J., and Yang, M.H. (2017). Face parsing via recurrent propagation. arXiv.10.5244/C.31.8 ↗
15. Guo, T., Kim, Y., Zhang, H., Qian, D., Yoo, B., Xu, J., Zou, D., Han., J., and Choi, C. (2018, January 2–7). Residual Encoder Decoder Network and Adaptive Prior for Face Parsing. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12268 ↗
16. Lin, J., Yang, H., Chen, D., Zeng, M., Wen, F., and Yuan, L. (2019, January 16–20). Face Parsing with RoI Tanh-Warping. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00580 ↗
17. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W., and Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size. arXiv.
18. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto., M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
19. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. (2018, January 19–21). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, MA, USA.10.1109/CVPR.2018.00474 ↗
20. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 19–21). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, MA, USA.10.1109/CVPR.2018.00716 ↗
21. Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8–14). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_8 ↗
22. Chen, Y., Fan, H., Xu, B., Yan, Z., Kalantidis, Y., Rohrbach, M., Yan, S., and Feng, J. (2019, January 16–20). Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. Proceedings of the IEEE International Conference on Computer Vision, Long Beach, CA, USA.10.1109/ICCV.2019.00353 ↗
23. Li, H., Xiong, P., Fan, H., and Sun, J. (2019, January 16–20). Dfanet: Deep feature aggregation for real-time semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00975 ↗
24. Li, G., Yun, I., Kim, J., and Kim, J. (2019). DABNet: Depth-wise Asymmetric Bottleneck for Real-time Semantic Segmentation. arXiv.
25. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
26. He, Y., Zhang, X., and Sun, J. (2017, January 22–19). Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.155 ↗
27. Yu, F., and Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv.
28. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
29. Zhang, F., Chen, Y., Li, Z., Hong, Z., Liu, J., Ma, F., Han., J., and Ding, E. (November, January 27). ACFNet: Attentional class feature network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
30. Zhang, Z., Zhang, X., Peng, C., Xue, X., and Sun, J. (2018, January 8–14). Exfuse: Enhancing feature fusion for semantic segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_17 ↗
31. Hu, J., Shen, L., and Sun, G. (2018, January 19–21). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, MA, USA.10.1109/CVPR.2018.00745 ↗
32. Woo, S., Park, J., Lee, J.Y., and So Kweon, I. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
33. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 16–20). Dual attention network for scene segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
34. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). Ccnet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
35. Ruan, T., Liu, T., Huang, Z., Wei, Y., Wei, S., and Zhao, Y. (February, January 27). Devil in the details: Towards accurate single and multiple human parsing. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
36. Zhang, Z., Fu, H., Dai, H., Shen, J., Pang, Y., and Shao, L. (2019, January 13–17). ET-Net: A Generic Edge-aTtention Guidance Network for Medical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32239-7_49 ↗
37. Si, H., Zhang, Z., Lv, F., Yu, G., and Lu, F. (2019). Real-Time Semantic Segmentation via Multiply Spatial Fusion Network. arXiv.
38. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation10.1109/TITS.2017.2750080 ↗
39. Karras, T., Laine, S., and Aila, T. (2019, January 16–20). A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗