Deep Learning Based Human Activity Recognition Using Spatio-Temporal Image Formation of Skeleton Joints
Source abstract
Abstract
Human activity recognition has become a significant research trend in the fields of computer vision, image processing, and human–machine or human–object interaction due to cost-effectiveness, time management, rehabilitation, and the pandemic of diseases. Over the past years, several methods published for human action recognition using RGB (red, green, and blue), depth, and skeleton datasets. Most of the methods introduced for action classification using skeleton datasets are constrained in some perspectives including features representation, complexity, and performance. However, there is still a challenging problem of providing an effective and efficient method for human action discrimination using a 3D skeleton dataset. There is a lot of room to map the 3D skeleton joint coordinates into spatio-temporal formats to reduce the complexity of the system, to provide a more accurate system to recognize human behaviors, and to improve the overall performance. In this paper, we suggest a spatio-temporal image formation (STIF) technique of 3D skeleton joints by capturing spatial information and temporal changes for action discrimination. We conduct transfer learning (pretrained models- MobileNetV2, DenseNet121, and ResNet18 trained with ImageNet dataset) to extract discriminative features and evaluate the proposed method with several fusion techniques. We mainly investigate the effect of three fusion methods such as element-wise average, multiplication, and maximization on the performance variation to human action recognition. Our deep learning-based method outperforms prior works using UTD-MHAD (University of Texas at Dallas multi-modal human action dataset) and MSR-Action3D (Microsoft action 3D), publicly available benchmark 3D skeleton datasets with STIF representation. We attain accuracies of approximately 98.93%, 99.65%, and 98.80% for UTD-MHAD and 96.00%, 98.75%, and 97.08% for MSR-Action3D skeleton datasets using MobileNetV2, DenseNet121, and ResNet18, respectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronics and Information Engineering, Korea Aerospace University, Goyang 10540, Korea
Department of Computer Science and Engineering, University of Chittagong, Chittagong 4331, Bangladesh
Funding
Gyeonggi province, Korea
References (68)
Show source-provided references
1. Continuous human action recognition using depth-MHI-HOG and a spotter model10.3390/s150305197 ↗
2. Dawar, N., and Kehtarnavaz, N. (2017, January 17–20). Continuous detection and recognition of actions of interest among actions of non-interest using a depth camera. Proceedings of the IEEE International Conference on Image Processing, Beijing, China.10.1109/ICIP.2017.8297079 ↗
3. Chu, X., Ouyang, W., Li, H., and Wang, X. (July, January 26). Structured feature learning for pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
4. Semantic human activity recognition: A literature review10.1016/j.patcog.2015.03.006 ↗
5. A vision-based system for intelligent monitoring: Human behaviour analysis and privacy by context10.3390/s140508895 ↗
6. Wei, H., Laszewski, M., and Kehtarnavaz, N. (2018, January 2–12). Deep Learning-Based Person Detection and Classification for Far Field Video Surveillance. Proceedings of the 13th IEEE Dallas Circuits and Systems Conference, Dallas, TX, USA.10.1109/DCAS.2018.8620111 ↗
7. Zhu, H., Vial, R., and Lu, S. (2017, January 22–29). Tornado: A spatio-temporal convolutional regression network for video action proposal. Proceedings of the CVPR, Venice, Italy.10.1109/ICCV.2017.619 ↗
8. Wen, R., Nguyen, B.P., Chng, C.B., and Chui, C.K. (2013, January 5–6). In Situ Spatial AR Surgical Planning Using projector-Kinect System. Proceedings of the Fourth Symposium on Information and Communication Technology, Da Nang, Vietnam.10.1145/2542050.2542060 ↗
9. A survey of augmented reality10.1162/pres.1997.6.4.355 ↗
10. A Depth Video Sensor-Based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments10.3390/s140711735 ↗
11. Unobtrusive Sensing and Wearable Devices for Health Informatics10.1109/TBME.2014.2309951 ↗
12. Survey of pedestrian action recognition techniques for autonomous driving10.26599/TST.2019.9010018 ↗
13. Bloom, V., Makris, D., and Argyriou, V. (2012, January 16–21). G3D: A gaming action dataset and real time action recognition evaluation framework. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.10.1109/CVPRW.2012.6239175 ↗
14. Kim, S.T., and Lee, H.J. (2020). Lightweight Stacked Hourglass Network for Human Pose Estimation. Appl. Sci., 10.10.3390/app10186497 ↗
15. Tasnim, N., Islam, M., and Baek, J.H. (2020). Deep Learning-Based Action Recognition Using 3D Skeleton Joints Information. Inventions, 5.10.3390/inventions5030049 ↗
16. Sun, Z., Liu, J., Ke, Q., and Rahmani, H. (2020). Human Action Recognition from Various Data Modalities: A Review. arXiv.
17. Pham, H.H., Salmane, H., Khoudour, L., Crouzil, A., Zegers, P., and Velastin, S.A. (2019). Spatio–temporal image representation of 3D skeletal movements for view-invariant action recognition with deep convolutional neural networks. Sensors, 19.10.20944/preprints201903.0086.v1 ↗
18. Human action recognition from RGB-D data using complete local binary pattern10.1016/j.cogsys.2019.05.002 ↗
19. Human action recognition using trajectory-based representation10.1016/j.eij.2015.05.002 ↗
20. Lowe, D.G. (1999, January 20–25). Object recognition from local scale-invariant features Computer Vision. Proceedings of the Seventh IEEE International Conference on Computer Vision, Corfu, Greece.10.1109/ICCV.1999.790410 ↗
21. Speeded-up robust features (SURF)10.1016/j.cviu.2007.09.014 ↗
22. Yang, X., Zhang, C., and Tian, Y. (2012, January 27–31). Recognizing actions using depth motion maps-based histograms of oriented gradients. Proceedings of the 20th ACM International Conference on Multimedia, Nara, Japan.10.1145/2393347.2396382 ↗
23. Oreifej, O., and Liu, Z. (2013, January 23–28). HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.10.1109/CVPR.2013.98 ↗
24. Mahjoub, A.B., and Atri, M. (2016, January 18–20). Human action recognition using RGB data. Proceedings of the 11th International Design & Test Symposium (IDT), Tunisia, Hammamet.
25. Al-Akam, R., and Paulus, D. (2018, January 16–18). Local and Global Feature Descriptors Combination from RGB-Depth Videos for Human Action Recognition. Proceedings of the ICPRAM, Funchal, Madeira, Portugal.10.5220/0006525002650272 ↗
26. Survey of convolutional neural network
27. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network10.1016/j.physd.2019.132306 ↗
28. Simonyan, K., and Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. arXiv.
29. Real-time action recognition with deeply transferred motion vector cnns10.1109/TIP.2018.2791180 ↗
30. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
31. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
32. Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 5–9). Action recognition from depth sequences using depth motion maps-based local binary patterns. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV.2015.150 ↗
33. Silhouette analysis-based action recognition via exploiting human poses10.1109/TCSVT.2012.2203731 ↗
34. Trelinski, J., and Kwolek, B. (2018, January 17–19). Convolutional Neural Network-Based Action Recognition on Depth Maps. Proceedings of the International Conference on Computer Vision and Graphics, Warsaw, Poland.10.1007/978-3-030-00692-1_19 ↗
35. Action recognition from depth maps using deep convolutional neural networks10.1109/THMS.2015.2504550 ↗
36. (2020, December 01). Torchvision Master. Available online: https://pytorch.org/docs/stable/torchvision/models.html.
37. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
38. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
39. Feichtenhofer, C., Pinz, A., and Wildes, R.P. (2016). Spatiotemporal residual networks for video action recognition. arXiv.10.1109/CVPR.2017.787 ↗
40. Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 27–30). UTD-MHAD: A Multimodal Dataset for Human Action Recognition Utilizing a Depth Camera and a Wearable Inertial Sensor. Proceedings of the IEEE International Conference on Image Processing, Quebec City, QC, Canada.10.1109/ICIP.2015.7350781 ↗
41. Li, W., Zhang, Z., and Liu, Z. (2010, January 13–18). Action recognition based on a bag of 3D points. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPRW.2010.5543273 ↗
42. Warchoł, D., and Kapuściński, T. (2020). Human Action Recognition Using Bone Pair Descriptor and Distance Descriptor. Symmetry, 12.10.3390/sym12101580 ↗
43. Vemulapalli, R., Arrate, F., and Chellappa, R. (2014, January 23–28). Human action recognition by representing 3d skeletons as points in a lie group. Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA.10.1109/CVPR.2014.82 ↗
44. Ke, Q., Bennamoun, M., An, S., Sohel, F., and Boussaid, F. (2017, January 21–26). A new representation of skeleton sequences for 3d action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA.10.1109/CVPR.2017.486 ↗
45. 3D skeleton-based human action classification: A survey10.1016/j.patcog.2015.11.019 ↗
46. Xia, L., Chen, C.C., and Aggarwal, J.K. (2012, January 16–21). View invariant human action recognition using histograms of 3d joints. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.10.1109/CVPRW.2012.6239233 ↗
47. Yang, X., and Tian, Y.L. (2012, January 16–21). Eigenjoints-based action recognition using naive-bayes-nearest-neighbor. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Providence, RI, USA.10.1109/CVPRW.2012.6239232 ↗
48. Zhu, Y., Chen, W., and Guo, G. (2013, January 23–28). Fusing spatiotemporal features and joints for 3d action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA.10.1109/CVPRW.2013.78 ↗
49. Evangelidis, G., Singh, G., and Horaud, R. (2014, January 24–28). Skeletal quads: Human action recognition using joint quadruples. Proceedings of the 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.772 ↗
50. Zanfir, M., Leordeanu, M., and Sminchisescu, C. (2013, January 1–8). The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection. Proceedings of the IEEE international conference on computer vision, Sydney, Australia.10.1109/ICCV.2013.342 ↗
51. Improving bag-of-poses with semi-temporal pose descriptors for skeleton-based action recognition10.1007/s00371-018-1489-7 ↗
52. Discriminative multi-instance multitask learning for 3D action recognition10.1109/TMM.2016.2626959 ↗
53. Du, Y., Wang, W., and Wang, L. (2015, January 8–10). Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
54. Wang, H., and Wang, L. (2017, January 21–26). Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.387 ↗
55. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., and Xie, X. (2016, January 12–17). Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10451 ↗
56. Liu, J., Shahroudy, A., Xu, D., and Wang, G. (2016, January 8–16). Spatio-temporal lstm with trust gates for 3d human action recognition. Proceedings of the European conference on computer vision, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_50 ↗
57. Song, S., Lan, C., Xing, J., Zeng, W., and Liu, J. (2017, January 4–9). An end-to-end spatio-temporal attention model for human action recognition from skeleton data. Proceedings of the AAAI conference on artificial intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11212 ↗
58. Li, C., Zhong, Q., Xie, D., and Pu, S. (2017, January 10–14). Skeleton-based action recognition with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong.
59. Skeleton-based action recognition with hierarchical spatial reasoning and temporal stack learning network10.1016/j.patcog.2020.107511 ↗
60. Zhang, S., Liu, X., and Xiao, J. (2017, January 24–31). On geometric features for skeleton-based action recognition using multilayer lstm networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.24 ↗
61. Du, Y., Fu, Y., and Wang, L. (2015, January 3–6). Skeleton based action recognition with convolutional neural network. Proceedings of the 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), Kualalumpur Westin Hotel Kuala Lumpur, Malaysia.10.1109/ACPR.2015.7486569 ↗
62. Liu, J., Akhtar, N., and Mian, A. (2019, January 16–20). Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition. Proceedings of the CVPR Workshops, Long Beach, CA, USA.
63. Ren, J., Reyes, N.H., Barczak, A.L.C., Scogings, C., and Liu, M. (2018, January 27–29). An investigation of skeleton-based optical flow-guided features for 3D action recognition using a multi-stream CNN model. Proceedings of the IEEE 3rd International Conference on Image, Vision and Computing (ICIVC), Chongqing, China.10.1109/ICIVC.2018.8492894 ↗
64. Li, B., Dai, Y., Cheng, X., Chen, H., Lin, Y., and He, M. (2017, January 10–14). Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong, China.
65. ConvNets-based action recognition from skeleton motion maps10.1007/s11042-019-08261-1 ↗
66. Joint distance maps based action recognition with convolutional neural networks10.1109/LSP.2017.2678539 ↗
67. Skeleton optical spectra-based action recognition using convolutional neural networks10.1109/TCSVT.2016.2628339 ↗
68. Wang, P., Li, Z., Hou, Y., and Li, W. (2016, January 15–19). Action recognition based on joint trajectory maps using convolutional neural networks. Proceedings of the 24th ACM international conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2967191 ↗