SCGFormer: Semantic Chebyshev Graph Convolution Transformer for 3D Human Pose Estimation
Source abstract
Abstract
With the rapid advancement of deep learning, 3D human pose estimation has largely freed itself from reliance on manually annotated methods. The effective utilization of joint features has become significant. Utilizing 2D human joint information to predict 3D human skeletons is of paramount importance. Effectively leveraging 2D joint data can improve the accuracy of 3D human skeleton prediction. In this paper, we propose the SCGFormer model to reduce the error in predicting human skeletal poses in three-dimensional space. The network architecture of SCGFormer encompasses Transformer and two distinct types of graph convolution, organized into two interconnected modules: SGraAttention and AcChebGconv. SGraAttention extracts global feature information from each 2D human joint, thereby augmenting local feature learning by integrating prior knowledge of human joint relationships. Simultaneously, AcChebGconv broadens the receptive field for graph structure information and constructs implicit joint relationships to aggregate more valuable adjacent features. SCGraFormer is tested on widely recognized benchmark datasets such as Human3.6M and MPI-INF-3DHP and achieves excellent results. In particular, on Human3.6M, our method achieves the best results in 9 actions (out of a total of 15 actions), with an overall average error reduction of about 1.5 points compared to state-of-the-art methods, demonstrating the excellent performance of SCGFormer.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Electronic Information, Guangxi University, Nanning 530004, China
Guangxi Key Laboratory of Multimedia Communications and Network Technology, Nanning 530004, China
Funding
Natural Science Foundation of China
References (61)
Show source-provided references
1. Staudemeyer, R.C., and Morris, E.R. (2019). Understanding LSTM—A tutorial into long short-term memory recurrent neural networks. arXiv.
2. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
3. Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y. (2013). Spectral networks and locally connected networks on graphs. arXiv.
4. Deep learning on graphs: A survey10.1109/TKDE.2020.2981333 ↗
5. In-air handwritten English word recognition using attention recurrent translator10.1007/s00521-017-3260-9 ↗
6. Lu, D., and Luo, L. (2020, January 14–19). Fmkit: An in-air-handwriting analysis library and data repository. Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Virtual.
7. Weng, J., Weng, C., and Yuan, J. (2017, January 21–26). Spatio-temporal naive-bayes nearest-neighbor (st-nbnn) for skeleton-based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.55 ↗
8. Yan, S., Xiong, Y., and Lin, D. (2018, January 2–7). Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12328 ↗
9. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019, January 15–20). Actional-structural graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00371 ↗
10. Jiang, S., Sun, B., Wang, L., Bai, Y., Li, K., and Fu, Y. (2021, January 19–25). Skeleton aware multi-modal sign language recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPRW53098.2021.00380 ↗
11. Pressure distribution image based human motion tracking system using skeleton and surface integration model10.1109/ROBOT.2001.933111 ↗
12. Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. (2017, January 10–12). Monocular 3d human pose estimation in the wild using improved cnn supervision. Proceedings of the 2017 International Conference on 3D Vision (3DV), Qingdao, China.10.1109/3DV.2017.00064 ↗
13. Pavlakos, G., Zhou, X., Derpanis, K.G., and Daniilidis, K. (2017, January 21–26). Coarse-to-fine volumetric prediction for single-image 3D human pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.139 ↗
14. Yang, W., Ouyang, W., Wang, X., Ren, J., Li, H., and Wang, X. (2018, January 18–23). 3d human pose estimation in the wild by adversarial learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00551 ↗
15. Zhou, X., Huang, Q., Sun, X., Xue, X., and Wei, Y. (2017, January 22–29). Towards 3d human pose estimation in the wild: A weakly-supervised approach. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.51 ↗
16. Lin, K., Wang, L., and Liu, Z. (2021, January 19–25). End-to-end human pose and mesh reconstruction with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00199 ↗
17. Tekin, B., Katircioglu, I., Salzmann, M., Lepetit, V., and Fua, P. (2016). Structured prediction of 3d human pose with deep neural networks. arXiv.10.5244/C.30.130 ↗
18. Sun, X., Shang, J., Liang, S., and Wei, Y. (2017, January 22–29). Compositional human pose regression. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.284 ↗
19. Toshev, A., and Szegedy, C. (2014, January 23–28). Deeppose: Human pose estimation via deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.214 ↗
20. Newell, A., Yang, K., and Deng, J. (2016, January 11–14). Stacked hourglass networks for human pose estimation. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part VIII 14.10.1007/978-3-319-46484-8_29 ↗
21. Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., and Sun, J. (2018, January 18–23). Cascaded pyramid network for multi-person pose estimation. Proceedings of the the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00742 ↗
22. Xiao, B., Wu, H., and Wei, Y. (2018, January 8–14). Simple baselines for human pose estimation and tracking. Proceedings of the the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01231-1_29 ↗
23. Osokin, D. (2018). Real-time 2d multi-person pose estimation on cpu: Lightweight openpose. arXiv.10.5220/0007555407440748 ↗
24. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15–20). Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00584 ↗
25. Ci, H., Wang, C., Ma, X., and Wang, Y. (November, January 27). Optimizing network structure for 3d human pose estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
26. Xu, T., and Takano, W. (2021, January 19–25). Graph stacked hourglass networks for 3d human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.01584 ↗
27. Zhao, L., Peng, X., Tian, Y., Kapadia, M., and Metaxas, D.N. (2019, January 15–20). Semantic graph convolutional networks for 3d human pose regression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00354 ↗
28. Zhao, W., Wang, W., and Tian, Y. (2022, January 19–24). GraFormer: Graph-oriented transformer for 3D pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01979 ↗
29. Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv.
30. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments10.1109/TPAMI.2013.248 ↗
31. Ionescu, C., Li, F., and Sminchisescu, C. (2011, January 6–13). Latent structured models for human pose estimation. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126500 ↗
32. Recent advances of monocular 2d and 3d human pose estimation: A deep learning perspective
33. 3d human pose estimation: A review of the literature and analysis of covariates10.1016/j.cviu.2016.09.002 ↗
34. Recovering 3D human pose from monocular images10.1109/TPAMI.2006.21 ↗
35. Ohashi, T., Ikegami, Y., Yamamoto, K., Takano, W., and Nakamura, Y. (2018, January 1–5). Video motion capture from the part confidence maps of multi-camera images by spatiotemporal filtering using the human skeletal model. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593867 ↗
36. Action database for categorizing and inferring human poses from video sequences10.1016/j.robot.2015.03.001 ↗
37. Martinez, J., Hossain, R., Romero, J., and Little, J.J. (2017, January 22–29). A simple yet effective baseline for 3d human pose estimation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.288 ↗
38. Wandt, B., and Rosenhahn, B. (2019, January 15–20). Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00797 ↗
39. Liu, J., Rojas, J., Li, Y., Liang, Z., Guan, Y., Xi, N., and Zhu, H. (2021, January 1–5). A graph attention spatio-temporal convolutional network for 3D human pose estimation in video. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Madrid, Spain.10.1109/ICRA48506.2021.9561605 ↗
40. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
41. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
42. Artificial neural networks (the multilayer perceptron)—A review of applications in the atmospheric sciences10.1016/S1352-2310(97)00447-0 ↗
43. Defferrard, M., Bresson, X., and Vandergheynst, P. (2016, January 5–8). Convolutional neural networks on graphs with fast localized spectral filtering. Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
44. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv.
45. Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., and Ding, Z. (2021, January 11–17). 3d human pose estimation with spatial and temporal transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Virtually.10.1109/ICCV48922.2021.01145 ↗
46. Li, W., Liu, H., Tang, H., Wang, P., and Van Gool, L. (2022, January 19–24). Mhformer: Multi-hypothesis transformer for 3d human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01280 ↗
47. Classes for fast maximum entropy training10.1109/ICASSP.2001.940893 ↗
48. Mikolov, T., Kombrink, S., Burget, L., Černockỳ, J., and Khudanpur, S. (2001, January 7–11). Extensions of recurrent neural network language model. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Salt Lake City, UT, USA.10.1109/ICASSP.2011.5947611 ↗
49. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
50. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
51. Dropout: A simple way to prevent neural networks from overfitting
52. Zhao, W., Tian, Y., Ye, Q., Jiao, J., and Wang, W. (2021). Graformer: Graph convolution transformer for 3d pose estimation. arXiv.10.1109/CVPR52688.2022.01979 ↗
53. Liu, K., Ding, R., Zou, Z., Wang, L., and Tang, W. (2020, January 23–28). A comprehensive study of weight sharing in graph networks for 3d human pose estimation. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part X 16.10.1007/978-3-030-58607-2_19 ↗
54. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
55. Pavlakos, G., Zhou, X., and Daniilidis, K. (2018, January 18–23). Ordinal depth supervision for 3d human pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00763 ↗
56. Sharma, S., Varigonda, P.T., Bindal, P., Sharma, A., and Jain, A. (November, January 27). Monocular 3d human pose estimation by generation and ordinal ranking. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
57. Fang, H.S., Xu, Y., Wang, W., Liu, X., and Zhu, S.C. (2018, January 2–7). Learning pose grammar to encode human body configuration for 3d pose estimation. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12270 ↗
58. Hossain, M.R.I., and Little, J.J. (2018, January 8–14). Exploiting temporal information for 3d human pose estimation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_5 ↗
59. Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 23–28). 2d human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.471 ↗
60. Luo, C., Chu, X., and Yuille, A. (2018). Orinet: A fully convolutional network for 3d human pose estimation. arXiv.
61. Zhou, K., Han, X., Jiang, N., Jia, K., and Lu, J. (November, January 27). Hemlets pose: Learning part-centric heatmap triplets for accurate 3d human pose estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.