Lightweight Stacked Hourglass Network for Human Pose Estimation
Source abstract
Abstract
Human pose estimation is a problem that continues to be one of the greatest challenges in the field of computer vision. While the stacked structure of an hourglass network has enabled substantial progress in human pose estimation and key-point detection areas, it is largely used as a backbone network. However, it also requires a relatively large number of parameters and high computational capacity due to the characteristics of its stacked structure. Accordingly, the present work proposes a more lightweight version of the hourglass network, which also improves the human pose estimation performance. The new hourglass network architecture utilizes several additional skip connections, which improve performance with minimal modifications while still maintaining the number of parameters in the network. Additionally, the size of the convolutional receptive field has a decisive effect in learning to detect features of the full human body. Therefore, we propose a multidilated light residual block, which expands the convolutional receptive field while also reducing the computational load. The proposed residual block is also invariant in scale when using multiple dilations. The well-known MPII and LSP human pose datasets were used to evaluate the performance using the proposed method. A variety of experiments were conducted that confirm that our method is more efficient compared to current state-of-the-art hourglass weight-reduction methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Video Surveillance and Tracking Methods · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Division of Computer Science and Engineering, Jeonbuk National University, Jeonju 54896, Korea
Funding
National Research Foundation of Korea
References (47)
Show source-provided references
1. Human pose estimation via convolutional part heatmap regression
2. Newell, A., Yang, K., and Deng, J. (2016, January 11–14). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
3. Tompson, J.J., Jain, A., LeCun, Y., and Bregler, C. (2014). Joint training of a convolutional network and a graphical model for human pose estimation. Advances in Neural Information Processing Systems, MIT Press.
4. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
5. Human pose estimation with deeply learned Multi-scale compositional models10.1109/ACCESS.2019.2919154 ↗
6. Chu, X., Yang, W., Ouyang, W., Ma, C., Yuille, A.L., and Wang, X. (2017, January 21–26). Multi-context Attention for Human Pose Estimation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.601 ↗
7. Peng, X., Tang, Z., Yang, F., Feris, R.S., and Metaxas, D. (2018, January 18–22). Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00237 ↗
8. Hierarchical binary CNNs for landmark localization with limited resources10.1109/TPAMI.2018.2866051 ↗
9. Yang, W., Li, S., Ouyang, W., Li, H., and Wang, X. (2017, January 22–29). Learning feature pyramids for human pose estimation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.144 ↗
10. Tang, W., and Wu, Y. (2019, January 15–21). Does learning specific features for related parts help human pose estimation?. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00120 ↗
11. Knowledge-guided deep fractal neural networks for human pose estimation10.1109/TMM.2017.2762010 ↗
12. Ke, L., Chang, M.-C., Qi, H., and Lyu, S. (2018, January 13–16). Multi-scale structure-aware network for human pose estimation. Proceedings of the Haptics: Science, Technology, Applications, Pisa, Italy.10.1109/ICIP.2018.8451114 ↗
13. Zhang, Y., Liu, J., and Huang, K. (2018). Dilated hourglass networks for human pose estimation. Chin. Autom. Congr., 2597–2602.10.1109/CAC.2018.8623582 ↗
14. Artacho, B., and Savakis, A. (2020, January 13–19). UniPose: Unified human pose estimation in single images and videos. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00706 ↗
15. Wei, S.-E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27–30). Convolutional pose machines. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.511 ↗
16. Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 23–28). 2D human pose estimation: New benchmark and state of the art analysis. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.471 ↗
17. Johnson, S., and Everingham, M. (September, January 31). Clustered pose and nonlinear appearance models for human pose estimation. Proceedings of the British Machine Vision Conference 2010, Aberystwyth, UK.10.5244/C.24.12 ↗
18. Bulat, A., and Tzimiropoulos, G. (2017, January 22–29). Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.400 ↗
19. Toshev, A., and Szegedy, D. (2014, January 23–28). DeepPose: Human pose estimation via deep neural networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.214 ↗
20. Yang, Y., and Ramanan, D. (2011, January 20–25). Articulated pose estimation with flexible mixtures-of-parts. Proceedings of the CVPR 2011, Providence, RI, USA.10.1109/CVPR.2011.5995741 ↗
21. Ferrari, V., Marín-Jiménez, M.J., and Zisserman, A. (2008, January 24–26). Progressive search space reduction for human pose estimation. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, USA.10.1109/CVPR.2008.4587468 ↗
22. Heterogeneous multi-task learning for human pose estimation with deep convolutional neural network10.1007/s11263-014-0767-8 ↗
23. Jain, A., Tompson, J., Andriluka, M., Taylor, G.W., and Bregler, C. (2013). Learning human pose estimation features with convolutional networks. arXiv.
24. Belagiannis, V., and Zisserman, A. (June, January 30). Recurrent human pose estimation. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.10.1109/FG.2017.64 ↗
25. Chu, X., Ouyang, W., Li, H., and Wang, X. (2016, January 27–30). Structured feature learning for pose estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.510 ↗
26. Yang, W., Ouyang, W., Li, H., and Wang, X. (2016, January 27–30). End-to-end learning of deformable mixture of parts and deep convolutional neural networks for human pose estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.335 ↗
27. Cao, Z., Šimon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
28. Noh, H., Hong, S., and Han, B. (2015, January 7–13). Learning deconvolution network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.178 ↗
29. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
30. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
31. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15–21). Deep High-Resolution Representation Learning for Human Pose Estimation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00584 ↗
32. Xiao, B., Wu, H., and Wei, Y. (2018, January 8–14). Simple Baselines for Human Pose Estimation and Tracking. Proceedings of the Haptics: Science, Technology, Applications, Munich, Germany.10.1007/978-3-030-01231-1_29 ↗
33. Yu, F., and Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv.
34. Chollet, F. (2017, January 21–26). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
35. Identity Mappings in Deep Residual Networks
36. Johnson, S., and Everingham, M. (2011, January 20–25). Learning effective human pose estimation from inaccurate annotation. Proceedings of the CVPR 2011, Providence, RI, USA.10.1109/CVPR.2011.5995318 ↗
37. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4–9). Automatic differentiation in pytorch. Proceedings of the NeurIPS Workshop, Long Beach, CA, USA.
38. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
39. Pishchulin, L., Andriluka, M., Gehler, P., and Schiele, B. (2013, January 1–8). Strong appearance and expressive spatial models for human pose estimation. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.433 ↗
40. Carreira, J., Agrawal, P., Fragkiadaki, K., and Malik, J. (2016, January 27–30). Human pose estimation with iterative error feedback. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.512 ↗
41. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., and Bregler, C. (2015, January 7–12). Efficient object localization using convolutional NETWORKS. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298664 ↗
42. Hu, P., and Ramanan, D. (2016, January 27–30). Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.604 ↗
43. Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., and Schiele, B. (2016, January 27–30). Deepcut: Joint subset partition and labeling for multi person pose estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.533 ↗
44. Human pose estimation using deep consensus voting
45. Chained Predictions Using Convolutional Neural Networks
46. Rafi, U., Leibe, B., Gall, J., Kostrikov, I., Wilson, R.C., Hancock, E.R., Smith, W.A.P., Pears, N.E., and Bors, A.G. (2016, January 19–22). An efficient convolutional network for human pose estimation. Proceedings of the British Machine Vision Conference 2016, York, UK.10.5244/C.30.109 ↗
47. Deepercut: A deeper, stronger, and faster multi-person pose estimation model