Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid Model
Source abstract
Abstract
Seniors who live alone at home are at risk of falling and injuring themselves and, thus, may need a mobile robot that monitors and recognizes their poses automatically. Even though deep learning methods are actively evolving in this area, they have limitations in estimating poses that are absent or rare in training datasets. For a lightweight approach, an off-the-shelf 2D pose estimation method, a more sophisticated humanoid model, and a fast optimization method are combined to estimate joint angles for 3D pose estimation. As a novel idea, the depth ambiguity problem of 3D pose estimation is solved by adding a loss function deviation of the center of mass from the center of the supporting feet and penalty functions concerning appropriate joint angle rotation range. To verify the proposed pose estimation method, six daily poses were estimated with a mean joint coordinate difference of 0.097 m and an average angle difference per joint of 10.017 degrees. In addition, to confirm practicality, videos of exercise activities and a scene of a person falling were filmed, and the joint angle trajectories were produced as the 3D estimation results. The optimized execution time per frame was measured at 0.033 s on a single-board computer (SBC) without GPU, showing the feasibility of the proposed method as a real-time system.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Diabetic Foot Ulcer Assessment and Management · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronics Engineering, Seunghak Campus, Dong-A University, Busan 49315, Republic of Korea
Funding
National Research Foundation of Korea
References (34)
Show source-provided references
1. Su, M., Hayati, D.W., Tseng, S., Chen, J., and Wei, H. (2020). Smart Care Using a DNN-Based Approach for Activities of Daily Living (ADL) Recognition. Appl. Sci., 11.10.3390/app11010010 ↗
2. Noreils, F.R. (2017). Inverse kinematics for a Humanoid Robot: A mix between closed form and geometric solutions. Tech. Rep., 1–31.
3. Joint-level vision-based ergonomic assessment tool for construction workers10.1061/(ASCE)CO.1943-7862.0001647 ↗
4. Rokbani, N., Casals, A., and Alimi, A.M. (2015). IK-FA, a new heuristic inverse kinematics solver using firefly algorithm. Comput. Intell. Appl. Model. Control, 369–395.10.1007/978-3-319-11017-2_15 ↗
5. Xu, J., Yu, Z., Ni, B., Yang, J., Yang, X., and Zhang, W. (2020, January 13–19). Deep kinematics analysis for monocular 3d human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00098 ↗
6. Li, J., Xu, C., Chen, Z., Bian, S., Yang, L., and Lu, C. (2021, January 19–25). Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00339 ↗
7. 3D human pose estimation: A review of the literature and analysis of covariates10.1016/j.cviu.2016.09.002 ↗
8. Monocular human pose estimation: A survey of deep learning-based methods10.1016/j.cviu.2019.102897 ↗
9. Deep 3D human pose estimation: A review10.1016/j.cviu.2021.103225 ↗
10. BabyPose: Real-time decoding of baby’s non-verbal communication using 2D video-based pose estimation10.1109/JSEN.2022.3183502 ↗
11. Vision-based human fall detection systems using deep learning: A review10.1016/j.compbiomed.2022.105626 ↗
12. Pavlakos, G., Zhou, X., Derpanis, K.G., and Daniilidis, K. (2017, January 21–26). Coarse-to-fine volumetric prediction for single-image 3D human pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.139 ↗
13. Luvizon, D.C., Picard, D., and Tabia, H. (2018, January 18–22). 2d/3d pose estimation and action recognition using multitask deep learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00539 ↗
14. Li, S., and Chan, A.B. (2014, January 1–5). 3d human pose estimation from monocular images with deep convolutional neural network. Proceedings of the Asian Conference on Computer Vision, Singapore.10.1007/978-3-319-16808-1_23 ↗
15. Zhou, X., Sun, X., Zhang, W., Liang, S., and Wei, Y. (2016, January 11–14). Deep kinematic pose regression. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-49409-8_17 ↗
16. Tome, D., Russell, C., and Agapito, L. (2017, January 21–26). Lifting from the deep: Convolutional 3d pose estimation from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.603 ↗
17. Wang, J., Huang, S., Wang, X., and Tao, D. (2019). Not all parts are created equal: 3D pose estimation by modelling bi-directional dependencies of body parts. arXiv.10.1109/ICCV.2019.00786 ↗
18. Wandt, B., and Rosenhahn, B. (2019, January 16–20). Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00797 ↗
19. Humaneva; Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion10.1007/s11263-009-0273-6 ↗
20. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments10.1109/TPAMI.2013.248 ↗
21. Pavllo, D., Feichtenhofer, C., Grangier, D., and Auli, M. (2019, January 16–20). 3D human pose estimation in video with temporal convolutions and semi-supervised training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00794 ↗
22. (2021, December 28). MediaPipe Pose. Available online: https://google.github.io/mediapipe/solutions/pose.html.
23. On load motor parameter identification using univariate dynamic encoding algorithm for searches (uDEAS)10.1109/TEC.2008.926068 ↗
24. (2021, August 01). Vicon. Available online: https://www.vicon.com/.
25. Vakanski, A., Jun, H.P., Paul, D., and Baker, R. (2018). A data set of human body movements for physical rehabilitation exercises. Data, 3.10.3390/data3010002 ↗
26. Bazarevsky, V., and Grishchenko, I. (2021, August 10). On-Device, Real-Time Body Pose Tracking with MediaPipe BlazePose, Google Research. Available online: https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html.
27. A kinematic notation for lower-pair mechanisms based on matrices10.1115/1.4011045 ↗
28. Motion and walking stabilization of humanoids using sensory reflex control10.5772/63116 ↗
29. On the global convergence of univariate dynamic encoding algorithm for searches (uDEAS)
30. Automatic detection of cracks in raw steel block using Gabor filter optimized by univariate dynamic encoding algorithm for searches (uDEAS)10.1016/j.ndteint.2009.01.007 ↗
31. Trajectory generation schemes for bipedal ascending and descending stairs using univariate dynamic encoding algorithm for searches (uDEAS)10.1007/s12555-010-0515-y ↗
32. Kim, J.-W., Ahn, H., Seo, H.C., and Lee, S.C. (2022). Optimization of Solar/Fuel Cell Hybrid Energy System Using the Combinatorial Dynamic Encoding Algorithm for Searches (cDEAS). Energies, 15.10.3390/en15082779 ↗
33. Goldberg, D.E. (1999). Genetic Algorithm in Search, Optimization and Machine Learning, Addison Wesley.
34. (2022, March 15). Size Korea. Available online: https://sizekorea.kr.