A CNN Model for Human Parsing Based on Capacity Optimization
Source abstract
Abstract
Although a state-of-the-art performance has been achieved in pixel-specific tasks, such as saliency prediction and depth estimation, convolutional neural networks (CNNs) still perform unsatisfactorily in human parsing where semantic information of detailed regions needs to be perceived under the influences of variations in viewpoints, poses, and occlusions. In this paper, we propose to improve the robustness of human parsing modules by introducing a depth-estimation module. A novel scheme is proposed for the integration of a depth-estimation module and a human-parsing module. The robustness of the overall model is improved with the automatically obtained depth labels. As another major concern, the computational efficiency is also discussed. Our proposed human parsing module with 24 layers can achieve a similar performance as the baseline CNN model with over 100 layers. The number of parameters in the overall model is less than that in the baseline model. Furthermore, we propose to reduce the computational burden by replacing a conventional CNN layer with a stack of simplified sub-layers to further reduce the overall number of trainable parameters. Experimental results show that the integration of two modules contributes to the improvement of human parsing without additional human labeling. The proposed model outperforms the benchmark solutions and the capacity of our model is better matched to the complexity of the task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong Kong 999077, China
Funding
Natural Science Foundation of China
References (63)
Show source-provided references
1. Lin, T., Maire, M., Belongie, S., and Hays, J. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
2. The pascal visual object classes challenge a retrospective10.1007/s11263-014-0733-5 ↗
3. Cordts, M., Omran, M., Ramos, S., and Rehfeld, T. (July, January 26). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
4. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (arXiv, 2018). Path aggregation network for instance segmentation, arXiv.10.1109/CVPR.2018.00913 ↗
5. Peng, C., Zhang, X., Yu, G., Luo, G., and Sun, J. (arXiv, 2017). Large Kernel Matters—Improve Semantic Segmentation by Global Convolutional Network, arXiv.10.1109/CVPR.2017.189 ↗
6. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (arXiv, 2017). Mask r-cnn, arXiv.10.1109/ICCV.2017.322 ↗
7. Chen, X., Mottaghi, R., Liu, X., and Fidler, S. (2014, January 24–27). Detect what you can: Detecting and representing objects using holistic models and body parts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.254 ↗
8. Xia, F., Wang, P., Chen, X., and Yuille, A. (2017, January 22–25). Joint Multi-person Pose Estimation and Semantic Part Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.644 ↗
9. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
10. Gan, C., Lin, M., Yang, Y., and de Melo, G. (2016, January 12–17). Concepts not alone: Exploring pairwise relationships for zero-shot video activity recognition. Proceedings of the AAAI 2016, Phoenix, AZ, USA.10.1609/aaai.v30i1.10466 ↗
11. Zhao, R., Ouyang, W., and Wang, X. (2013, January 25–27). Unsupervised salience learning for person re-identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.460 ↗
12. Turban, E., King, D., Lee, J., and Viehland, D. (2002). Electronic Commerce: A Managerial Perspective 2002, Prentice Hall.
13. Xu, N., Price, B., Cohen, S., Yang, J., and Huang, T. (July, January 26). Deep interactive object selection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vega, NV, USA.
14. Collins, R., Lipton, A., and Kanade, T. (2000). A System for Video Surveillance and Monitoring, Robotics Institute, Carnegie Mellon University. VSAM Final Report.
15. Lin, J., Guo, X., Shao, J., Jiang, C., and Zhu, Y. (2016, January 24–28). A virtual reality platform for dynamic human-scene interaction. Proceedings of the SIGGRAPH, San Jose, CA, USA.10.1145/2992138.2992144 ↗
16. He, K., Zhang, X., Ren, S., and Sun, J. (Comput. Vis. Pattern Recognit. arXiv, 2016). Deep residual learning for image recognition, Comput. Vis. Pattern Recognit. arXiv.10.1109/CVPR.2016.90 ↗
17. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
18. Bearman, A., Russakovsky, O., Ferrari, V., and Li, F. (2016, January 8–16). What’s the point: Semantic segmentation with point supervision. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46478-7_34 ↗
19. Pathak, D., Krahenbuhl, P., and Darrell, T. (2015, January 11–18). Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.209 ↗
20. Dai, J., He, K., and Sun, J. (2015, January 11–18). Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.191 ↗
21. Papandreou, G., Chen, L., Murphy, K., and Yuille, A. (arXiv, 2015). Weakly-and semi-supervised learning of a DCNN for semantic image segmentation, arXiv.10.1109/ICCV.2015.203 ↗
22. Sun, J., Lin, D., Dai, J., Jia, J., and He, K. (July, January 26). Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
23. Photo tourism: Exploring photo collections in 3D10.1145/1141911.1141964 ↗
24. Snavely, N., and Li, Z. (2018, January 19–21). MegaDepth: Learning Single-View Depth Prediction from Internet Photos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.
25. Newell, A., Yang, K., and Deng, J. (2016, January 8–16). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
26. Liang, X., Gong, K., Shen, X., and Lin, L. (arXiv, 2018). Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmark, arXiv.10.1109/TPAMI.2018.2820063 ↗
27. Ierusalem, A. (arXiv, 2018). Catastrophic Importance of Catastrophic Forgetting, arXiv.
28. He, Y., Lin, J., Liu, Z., and Wang, H. (2018, January 8–14). Amc: Automl for model compression and acceleration on mobile devices. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_48 ↗
29. (2019). Shallow learning for deep networks. Int. Conf. Learn. Represent., Under review.
30. Huang, G., Liu, Z., and Van Der Maaten, L. (2017, January 22–25). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
31. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
32. Pohlen, T., Hermans, A., Mathias, M., and Leibe, B. (arXiv, 2017). Full-resolution residual networks for semantic segmentation in street scenes, arXiv.10.1109/CVPR.2017.353 ↗
33. Chen, L., George, P., Florian, S., and Hartwig, A. (arXiv, 2017). Rethinking atrous convolution for semantic image segmentation, arXiv.
34. Zhao, J., Li, J., Cheng, Y., Zhou, L., and Sim, T. (arXiv, 2018). Understanding Humans in Crowded Scenes: Deep Nested Adversarial Learning and A New Benchmark for Multi-Human Parsing, arXiv.10.1145/3240508.3240509 ↗
35. Liu, S., Wang, C., Qian, R., Yu, H., and Bao, R. (2017, January 22–25). Surveillance video parsing with single frame supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.114 ↗
36. Zhao, J., Li, J., Nie, X., Zhao, F., Chen, Y., Wang, Z., Feng, J., and Yan, S. (2017, January 22–25). Self-supervised neural aggregation networks for human parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPRW.2017.204 ↗
37. Li, Q., Arnab, A., and Torr, P. (arXiv, 2017). Holistic, instance-level human parsing, arXiv.10.5244/C.31.25 ↗
38. Deepcut: Object segmentation from bounding box annotations using convolutional neural networks10.1109/TMI.2016.2621185 ↗
39. Çiçek, Ö., Abdulkadir, A., Lienkamp, S., and Brox, T. (2016, January 17–21). 3D U-Net: Learning dense volumetric segmentation from sparse annotation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece.10.1007/978-3-319-46723-8_49 ↗
40. Vezhnevets, A., and Buhmann, J. (2010, January 13–18). Towards weakly supervised semantic segmentation by means of multiple instance and multi-task learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5540060 ↗
41. Xu, J., Schwing, A.G., and Urtasun, R. (2015, January 7–12). Learning to segment under various forms of weak supervision. Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299002 ↗
42. Ha, D., Dai, A., and Le, Q. (arXiv, 2016). Hypernetworks, arXiv.
43. Elhoseiny, M., Saleh, B., and Elgammal, A. (2013, January 1–8). Write a classifier: Zero-shot learning using purely textual descriptions. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.321 ↗
44. Wang, Y., and Hebert, M. (2016, January 8–16). Learning to learn: Model regression networks for easy small sample learning. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46466-4_37 ↗
45. Hu, R., Dollár, P., He, K., and Darrell, T. (arXiv, 2017). Learning to Segment Every Thing, arXiv.10.1109/CVPR.2018.00445 ↗
46. Hoffman, J., Guadarrama, S., Tzeng, E., and Hu, R. (2014, January 8–12). LSDA: Large scale detection through adaptation. Proceedings of the Advances in Neural Information Processing Systems, Montreal, Canada.
47. Zamir, A., Sax, A., Shen, W., Guibas, L., and Malik, J. (2018, January 19–21). Taskonomy: Disentangling Task Transfer Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00391 ↗
48. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
49. He, Y., Zhang, X., and Sun, J. (2017, January 22–25). Channel pruning for accelerating very deep neural networks. Proceedings of the International Conference on Computer Vision, Honolulu, HI, USA.10.1109/ICCV.2017.155 ↗
50. Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (arXiv, 2018). Shufflenet v2: Practical guidelines for efficient cnn architecture design, arXiv.10.1007/978-3-030-01264-9_8 ↗
51. Sandler, M., Howard, A., Zhu, M., and Zhmoginov, A. (2018, January 19–21). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
52. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
53. Chen, W., Fu, Z., Yang, D., and Deng, J. (2016, January 5–10). Single-image depth perception in the wild. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
54. Szegedy, C., Liu, W., Jia, Y., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
55. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the ICLR 2015, San Diego, CA, USA.
56. Oliveira, G., Valada, A., Bollen, C., and Burgard, W. (2016, January 17–20). Deep Learning for human part discovery in images. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487304 ↗
57. Chen, L., Yang, Y., Wang, J., and Yuille, A. (July, January 26). Attention to scale: Scale-aware semantic image segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
58. Xia, F., Wang, P., Chen, L.C., and Yuille, A. (2016, January 8–16). Zoom better to see clearer: Human part segmentation with auto zoom net. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.
59. Liang, X., Shen, X., Xiang, D., Feng, J., and Lin, L. (July, January 26). Semantic object parsing with local-global long short-term memory. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
60. Deng, J., Berg, A., Satheesh, S., and Su, H. (2012, September 01). Large Scale Visual Recognition Challenge 2012 (ILSVRC2012). Available online: http://www.image-net.org/challenges/LSVRC/2012/.
61. Lei, N., Luo, Z., Yau, S., and Gu, D. (2018). Geometric Understanding of Deep Learning. arXiv.
62. Jiang, Y., and Chi, Z. (2018). A CNN Model for Semantic Person Part Segmentation with Capacity Optimization. IEEE Trans. Image Process.10.1109/TIP.2018.2886785 ↗
63. Tobin, J., Fong, R., Ray, A., and Schneider, J. (2017, January 24–28). Domain Randomization for Transferring Deep Neural Networks from simulation to the real world. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202133 ↗