Biomac3D: 2D-to-3D Human Pose Analysis Model for Tele-Rehabilitation Based on Pareto Optimized Deep-Learning Architecture
Source abstract
Abstract
The research introduces a unique deep-learning-based technique for remote rehabilitative analysis of image-captured human movements and postures. We present a ploninomial Pareto-optimized deep-learning architecture for processing inverse kinematics for sorting out and rearranging human skeleton joints generated by RGB-based two-dimensional (2D) skeleton recognition algorithms, with the goal of producing a full 3D model as a final result. The suggested method extracts the entire humanoid character motion curve, which is then connected to a three-dimensional (3D) mesh for real-time preview. Our method maintains high joint mapping accuracy with smooth motion frames while ensuring anthropometric regularity, producing a mean average precision (mAP) of 0.950 for the task of predicting the joint position of a single subject. Furthermore, the suggested system, trained on the MoVi dataset, enables a seamless evaluation of posture in a 3D environment, allowing participants to be examined from numerous perspectives using a single recorded camera feed. The results of evaluation on our own self-collected dataset of human posture videos and cross-validation on the benchmark MPII and KIMORE datasets are presented.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Hand Gesture Recognition Systems · Stroke Rehabilitation and Recovery
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Multimedia Engineering, Faculty of Informatics, Kaunas University of Technology, 51368 Kaunas, Lithuania
Department of Biomechanical Engineering, Vilnius Gediminas Technical University, 03224 Vilnius, Lithuania
Department of Rehabilitation, Physical and Sports Medicine, Faculty of Medicine, Vilnius University, 08661 Vilnius, Lithuania
Funding
EU Structural Funds project financed by the European Regional Development Fund · Promotion of Centres of Excellence and Centres for Innovation and Technology Transfer
References (100+)
Show source-provided references
1. A data-centric artificial intelligent and extended reality technology in smart healthcare systems10.1007/s13278-022-00888-7 ↗
2. Ankle Rehabilitation Robotic Systems for domestic use—A systematic review10.1515/cdbme-2022-1018 ↗
3. Compliance with Upper Limb Home-Based Exergaming Interventions for Stroke Patients: A Narrative Review10.2340/jrm.v54.2270 ↗
4. Technologies used to facilitate remote rehabilitation of adults with deconditioning, musculoskeletal conditions, stroke, or traumatic brain injury: An umbrella review10.11124/JBIES-21-00241 ↗
5. Malleret, T., and Schwab, K. (2020). COVID-19, ISBN Agentur Schweiz.
6. Advances in upper limb stroke rehabilitation: A technology push10.1007/s11517-011-0797-0 ↗
7. Head-Mounted Display-Based Virtual Reality and Physiological Computing for Stroke Rehabilitation: A Systematic Review10.3389/frvir.2022.889271 ↗
8. Effect of home-based virtual reality training and telerehabilitation on balance in individuals with Parkinson disease, multiple sclerosis, and stroke: A systematic review and meta-analysis10.1007/s10072-021-05855-2 ↗
9. Avers, D. (2020). Functional performance measures and assessment for older adults. Guccione’s Geriatric Physical Therapy, Elsevier.10.1016/B978-0-323-60912-8.00007-5 ↗
10. Telerehabilitation: Review of the State-of-the-Art and Areas of Application10.2196/rehab.7511 ↗
11. Alexander, M. (2022). Telerehabilitation, Elsevier—Health Sciences Division.
12. European Framework of Rehabilitation Services Types: The perspective of the Physical and Rehabilitation Medicine Section and Board of the European Union of Medical Specialists10.23736/S1973-9087.19.05728-9 ↗
13. Home-based technologies for stroke rehabilitation: A systematic review10.1016/j.ijmedinf.2018.12.001 ↗
14. Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities10.1016/j.jbiomech.2018.09.009 ↗
15. Application of OptiTrack motion capture systems in human movement analysis
16. The future of General Movement Assessment: The role of computer vision and machine learning—A scoping review10.1016/j.ridd.2021.103854 ↗
17. Kinematics-vis: A Visualization Tool for the Mathematics of Human Motion10.21105/joss.03490 ↗
18. Analysis and Quantification of Repetitive Motion in Long-Term Rehabilitation10.1109/JBHI.2018.2848103 ↗
19. Real-time inverse kinematics and inverse dynamics for lower limb applications using OpenSim10.1080/10255842.2016.1240789 ↗
20. Ziziene, J., Daunoraviciene, K., Juskeniene, G., and Raistenskis, J. (2022). Comparison of kinematic parameters of children gait obtained by inverse and direct models. PLoS ONE, 17.10.1371/journal.pone.0270423 ↗
21. Shippen, J., and May, B. (2016, January 20–22). BoB—Biomechanics in MATLAB. Proceedings of the 11th International Conference Biomdlore 2016, Druskininkai, Lithuania.10.3846/biomdlore.2016.02 ↗
22. Hribernik, M., Umek, A., Tomažič, S., and Kos, A. (2022). Review of Real-Time Biomechanical Feedback Systems in Sport and Rehabilitation. Sensors, 22.10.3390/s22083006 ↗
23. Using Biometric Technology for Telehealth and Telerehabilitation10.1016/j.pmr.2020.12.007 ↗
24. EHealth Solutions for the Integrated Healthcare10.1155/2018/3846892 ↗
25. A Control and Posture Recognition Strategy for Upper-Limb Rehabilitation of Stroke Patients10.1155/2021/6630492 ↗
26. IMU-Based Estimation of Lower Limb Motion Trajectory With Graph Convolution Network10.1109/JSEN.2021.3115105 ↗
27. Multiple Kinect based system to monitor and analyze key performance indicators of physical training10.1186/s13673-020-00256-4 ↗
28. Chang, W.J., Su, J.P., Chen, L.B., Hsu, C.H., Lin, C.P., Yang, T.C., Chen, M.C., and Ou, Y.K. (2019, January 15–18). BodyTracker: A deep learning based 3D limb trajectory tracking system for rehabilitation. Proceedings of the 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), Osaka, Japan.10.1109/GCCE46687.2019.9015359 ↗
29. Kinect and wearable inertial sensors for motor rehabilitation programs at home: State of the art and an experimental comparison10.1186/s12938-020-00762-7 ↗
30. Cerebral Infarction Rehabilitation Evaluation with Posture Analyses10.1088/1757-899X/612/2/022082 ↗
31. A system for head-neck rehabilitation exercises based on serious gaming and virtual reality10.1007/s11042-017-5328-z ↗
32. An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications10.1109/TNSRE.2020.2968912 ↗
33. Piraintorn, P., and Sa-ing, V. (2020, January 24–27). Stroke rehabilitation based on intelligence interaction system. Proceedings of the 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Phuket, Thailand.10.1109/ECTI-CON49241.2020.9158104 ↗
34. Lin, C.Y., Chen, P.Y., Bai, P.Z., Xu, Z.X., Liao, W.X., and Tsai, C.L. (2019, January 10–12). A mechanism for solving the elderly posture problem. Proceedings of the 2019 IEEE International Conference on Smart Cloud (SmartCloud), Tokyo, Japan.10.1109/SmartCloud.2019.000-2 ↗
35. A Motion Capturing and Energy Harvesting Hybridized Lower-Limb System for Rehabilitation and Sports Applications10.1002/advs.202101834 ↗
36. Passon, A., Schauer, T., and Seel, T. (2018, January 26–29). Hybrid inertial-robotic motion tracking for posture biofeedback in upper limb rehabilitation*. Proceedings of the 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), Enschede, The Netherlands.10.1109/BIOROB.2018.8487203 ↗
37. Multi-Task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
38. Web based Augmented Reality for Human Body Anatomy Learning10.1016/j.procs.2018.08.197 ↗
39. Detection of sitting posture using hierarchical image composition and deep learning10.7717/peerj-cs.442 ↗
40. Ogundokun, R.O., Maskeliūnas, R., and Damaševičius, R. (2022). Human Posture Detection Using Image Augmentation and Hyperparameter-Optimized Transfer Learning Algorithms. Appl. Sci., 12.10.3390/app121910156 ↗
41. Sitsen: Passive sitting posture sensing based on wireless devices10.1177/15501477211024846 ↗
42. Kulikajevas, A., Maskeliunas, R., Damasevicius, R., and Scherer, R. (2021). Humannet—A two-tiered deep neural network architecture for self-occluding humanoid pose reconstruction. Sensors, 21.10.3390/s21123945 ↗
43. A review of 3D human pose estimation algorithms for markerless motion capture10.1016/j.cviu.2021.103275 ↗
44. A Review of the Evolution of Vision-Based Motion Analysis and the Integration of Advanced Computer Vision Methods Towards Developing a Markerless System10.1186/s40798-018-0139-y ↗
45. 3D Tracking of Human Motion Using Visual Skeletonization and Stereoscopic Vision10.3389/fbioe.2020.00181 ↗
46. Demystification of AI-driven medical image interpretation: Past, present and future10.1007/s00330-018-5674-x ↗
47. Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T.L., Zhang, F., and Grundmann, M. (2020). BlazePose: On-device Real-time Body Pose tracking. arXiv.
48. LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals10.1038/s41592-021-01226-z ↗
49. Zhang, Z., Tang, J., and Wu, G. (2019). Simple and Lightweight Human Pose Estimation. arXiv.
50. Luo, Z., Hachiuma, R., Yuan, Y., and Kitani, K. (2021, January 6–14). Dynamics-regulated kinematic policy for egocentric pose estimation. Proceedings of the Advances in Neural Information Processing Systems, Virtual.
51. Learning to Estimate 3D Human Pose From Point Cloud10.1109/JSEN.2020.2999849 ↗
52. OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association10.1109/TITS.2021.3124981 ↗
53. Liu, Y., Xu, Y., and Li, S.b. (2018, January 25–27). 2-D human pose estimation from images based on deep learning: A review. Proceedings of the 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Xi’an, China.10.1109/IMCEC.2018.8469573 ↗
54. Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. (2014, January 15–17). 2D human pose estimation: New benchmark and state of the art analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New York, NY, USA.10.1109/CVPR.2014.471 ↗
55. A review of deep learning techniques for 2D and 3D human pose estimation10.1016/j.imavis.2021.104282 ↗
56. Tang, W., Yu, P., and Wu, Y. (2018, January 8–14). Deeply learned compositional models for human pose estimation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01219-9_12 ↗
57. Wei, S.E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27–30). Convolutional pose machines. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.511 ↗
58. Martinez, J., Hossain, R., Romero, J., and Little, J.J. (2017, January 22–29). A simple yet effective baseline for 3d human pose estimation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.288 ↗
59. Yan, S., Xiong, Y., Wang, J., and Lin, D. (2022, November 01). MMSkeleton. Available online: https://github.com/open-mmlab/mmskeleton.
60. Tome, D., Russell, C., and Agapito, L. (2017, January 21–26). Lifting from the deep: Convolutional 3D pose estimation from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.603 ↗
61. Zou, Z., and Tang, W. (2021, January 10–17). Modulated graph convolutional network for 3D human pose estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01128 ↗
62. Pavllo, D., Feichtenhofer, C., Grangier, D., and Auli, M. (2019, January 15–20). 3D human pose estimation in video with temporal convolutions and semi-supervised training. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00794 ↗
63. Moon, G., Chang, J., and Lee, K.M. (2018, January 18–20). V2V-PoseNet: Voxel-to-Voxel prediction network for accurate 3D hand and human pose estimation from a single depth map. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA.10.1109/CVPR.2018.00533 ↗
64. Martínez-González, A., Villamizar, M., Canévet, O., and Odobez, J.M. (2020–24, January 24). Residual pose: A decoupled approach for depth-based 3D human pose estimation. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV, USA.10.1109/IROS45743.2020.9340695 ↗
65. Li, J., Xu, C., Chen, Z., Bian, S., Yang, L., and Lu, C. (2021, January 20–25). HybrIK: A hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00339 ↗
66. Trumble, M., Gilbert, A., Hilton, A., and Collomosse, J. (2018, January 8–14). Deep autoencoder for combined human pose estimation and body model upscaling. Proceedings of the European Conference on Computer Vision (ECCV’18), Munich, Germany.10.1007/978-3-030-01249-6_48 ↗
67. Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos
68. Gärtner, E., Pirinen, A., and Sminchisescu, C. (2020, January 7–12). Deep reinforcement learning for active human pose estimation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6714 ↗
69. Sengupta, A., Budvytis, I., and Cipolla, R. (2021, January 10–17). Hierarchical kinematic probability distributions for 3D human shape and pose estimation from images in the wild. Proceedings of the International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01103 ↗
70. Cheng, Y., Yang, B., Wang, B., Wending, Y., and Tan, R. (November, January 27). Occlusion-Aware networks for 3D human pose estimation in video. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00081 ↗
71. Li, W., Liu, H., Tang, H., Wang, P., and Van Gool, L. (2021). MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation. arXiv.10.1109/CVPR52688.2022.01280 ↗
72. Ma, H., Chen, L., Kong, D., Wang, Z., Liu, X., Tang, H., Yan, X., Xie, Y., Lin, S.Y., and Xie, X. (2021, January 22–25). TransFusion: Cross-view fusion with transformer for 3D human pose estimation. Proceedings of the British Machine Vision Conference, Online.
73. Dwivedi, S.K., Athanasiou, N., Kocabas, M., and Black, M.J. (2021, January 11–17). Learning To regress bodies from images using differentiable semantic rendering. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01106 ↗
74. Shanyan, G., Jingwei, X., Yunbo, W., Bingbing, N., and Xiaokang, Y. (2021, January 20–25). Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.
75. Izatt, G., Dai, H., and Tedrake, R. (2019). Globally Optimal Object Pose Estimation in Point Clouds with Mixed-Integer Programming. Robotics Research, Springer International Publishing. Springer Proceedings in Advanced Robotics.10.1007/978-3-030-28619-4_49 ↗
76. Image Co-Saliency Detection and Co-Segmentation via Progressive Joint Optimization10.1109/TIP.2018.2861217 ↗
77. Liu, K., Zhao, Y., Nie, Q., Gao, Z., and Chen, B.M. (2022, January 23–27). Weakly supervised 3D scene segmentation with region-level boundary awareness and instance discrimination. Proceedings of the Computer Vision—ECCV 2022, Tel Aviv, Israel. Lecture Notes in Computer Science.10.1007/978-3-031-19815-1_3 ↗
78. Lippi, V., Maurer, C., and Mergner, T. (2022). Human-likeness indicator for robot posture control and balance. Communications in Computer and Information Science, Springer International Publishing.10.1007/978-3-031-19650-8_5 ↗
79. A Kinematic Notation for Lower-Pair Mechanisms Based on Matrices10.1115/1.4011045 ↗
80. Online human-like redundancy optimization for tele-operated anthropomorphic manipulators10.1177/1729881418814695 ↗
81. Papadopoulos, K., Demisse, G., Ghorbel, E., Antunes, M., Aouada, D., and Ottersten, B. (2019). Localized Trajectories for 2D and 3D Action Recognition. Sensors, 19.10.3390/s19163503 ↗
82. Advances in Remote Monitoring for Stroke Recovery10.1161/STROKEAHA.122.038885 ↗
83. Georgiadis, C., Karvounis, E., Koritsoglou, K., Votis, K., Tzovaras, D., Dimopoulos, D., Varvarousis, D., and Plouims, A. (2021, January 24–26). A remote rehabilitation training system using Virtual Reality. Proceedings of the 2021 6th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM), Preveza, Greece.10.1109/SEEDA-CECNSM53056.2021.9566227 ↗
84. Efficiency of physical therapy on postural imbalance after stroke: Study protocol for a systematic review and meta-analysis10.1136/bmjopen-2016-013348 ↗
85. Effects of visual feedback therapy on postural control in bilateral standing after stroke: A systematic review10.1080/16501970500344902 ↗
86. Ghorbani, S., Mahdaviani, K., Thaler, A., Körding, K.P., Cook, D.J., Blohm, G., and Troje, N.F. (2020). MoVi: A Large Multipurpose Motion and Video Dataset. arXiv.10.1371/journal.pone.0253157 ↗
87. A review of computational approaches for evaluation of rehabilitation exercises10.1016/j.compbiomed.2020.103687 ↗
88. Efficacy of Home-Based Telerehabilitation vs In-Clinic Therapy for Adults After Stroke10.1001/jamaneurol.2019.1604 ↗
89. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common objects in context. Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
90. Jung, H.Y., Lee, S., Heo, Y.S., and Yun, I.D. (2015, January 7–12). Random tree walk toward instantaneous 3D human pose estimation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
91. Gu, Y., Zhang, H., and Kamijo, S. (2020). Multi-Person Pose Estimation using an Orientation and Occlusion Aware Deep Learning Network. Sensors, 20.10.3390/s20061593 ↗
92. Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. (2017, January 10–12). Monocular 3D human pose estimation in the wild using improved CNN supervision. Proceedings of the 2017 Fifth International Conference on 3D Vision (3DV), Qingdao, China.10.1109/3DV.2017.00064 ↗
93. A Subject-Specific Kinematic Model to Predict Human Motion in Exoskeleton-Assisted Gait10.3389/fnbot.2018.00018 ↗
94. Spectral elements and field separation for an acoustic fluid subject to cavitation10.1016/S0021-9991(02)00024-4 ↗
95. Statistical methods for assessing agreement between two methods of clinical measurement10.1016/j.ijnurstu.2009.10.001 ↗
96. A Concordance Correlation Coefficient to Evaluate Reproducibility10.2307/2532051 ↗
97. Intraclass correlations: Uses in assessing rater reliability10.1037/0033-2909.86.2.420 ↗
98. Vabalas, A., Gowen, E., Poliakoff, E., and Casson, A.J. (2019). Machine learning algorithm validation with a limited sample size. PLoS ONE, 14.10.1371/journal.pone.0224365 ↗
99. The KIMORE Dataset: KInematic Assessment of MOvement and Clinical Scores for Remote Monitoring of Physical REhabilitation10.1109/TNSRE.2019.2923060 ↗
100. Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises10.1109/TNSRE.2022.3150392 ↗