A Comprehensive Review on Handcrafted and Learning-Based Action Representation Approaches for Human Activity Recognition
Source abstract
Abstract
Human activity recognition (HAR) is an important research area in the fields of human perception and computer vision due to its wide range of applications. These applications include: intelligent video surveillance, ambient assisted living, human computer interaction, human-robot interaction, entertainment, and intelligent driving. Recently, with the emergence and successful deployment of deep learning techniques for image classification, researchers have migrated from traditional handcrafting to deep learning techniques for HAR. However, handcrafted representation-based approaches are still widely used due to some bottlenecks such as computational complexity of deep learning techniques for activity recognition. However, approaches based on handcrafted representation are not able to handle complex scenarios due to their limitations and incapability; therefore, resorting to deep learning-based techniques is a natural option. This review paper presents a comprehensive survey of both handcrafted and learning-based action representations, offering comparison, analysis, and discussions on these approaches. In addition to this, the well-known public datasets available for experimentations and important applications of HAR are also presented to provide further insight into the field. This is the first review paper of its kind which presents all these aspects of HAR in a single review article with comprehensive coverage of each part. Finally, the paper is concluded with important discussions and research directions in the domain of HAR.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Context-Aware Activity Recognition Systems · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computing and Communications Infolab21, Lancaster University, Lancaster LA1 4WA, UK
Department of Computer Science, COMSATS Institute of Information Technology, Lahore 54000, Pakistan
References (100+)
Show source-provided references
1. Human Activity Analysis: A Review10.1145/1922649.1922653 ↗
2. Traditional and recent approaches in background modeling for foreground detection: An overview10.1016/j.cosrev.2014.04.001 ↗
3. A review on video-based human activity recognition10.3390/computers2020088 ↗
4. Human action recognition with video data: Research and evaluation challenges10.1109/THMS.2014.2325871 ↗
5. A survey on vision-based human action recognition10.1016/j.imavis.2009.11.014 ↗
6. A survey of vision-based methods for action representation, segmentation and recognition10.1016/j.cviu.2010.10.002 ↗
7. Semantic human activity recognition: A literature review10.1016/j.patcog.2015.03.006 ↗
8. Maravelakis, E., Konstantaras, A., Kilty, J., Karapidakis, E., and Katsifarakis, E. (2014, January 28–29). Automatic building identification and features extraction from aerial images: Application on the historic 1866 square of Chania Greece. Proceedings of the 2014 International Symposium on Fundamentals of Electrical Engineering (ISFEE), Bucharest, Romania.10.1109/ISFEE.2014.7050594 ↗
9. A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments10.3390/s140711735 ↗
10. Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart home10.1177/1420326X12469714 ↗
11. Building recognition using local oriented features10.1109/TII.2013.2245910 ↗
12. Jalal, A., Kamal, S., and Kim, D. (2015, January 25–27). Shape and motion features approach for activity tracking and recognition from kinect video camera. Proceedings of the 2015 IEEE 29th International Conference on Advanced Information Networking and Applications Workshops (WAINA), Gwangju, Korea.10.1109/WAINA.2015.38 ↗
13. Deep learning10.1038/nature14539 ↗
14. Yuan, R., and Hui, W. (2008, January 20–22). Object identification and recognition using multiple contours based moment invariants. Proceedings of the 2008 International Symposium on Information Science and Engineering, Shanghai, China.
15. Jalal, A., and Rasheed, Y.A. (2007, January 26–28). Collaboration achievement along with performance maintenance in video streaming. Proceedings of the IEEE Conference on Interactive Computer Aided Learning, Villach, Austria.
16. Subsiding OOB Emission and ICI Power Using iPOWER Pulse in OFDM Systems10.4316/AECE.2016.01011 ↗
17. Dense RGB-D map-based human tracking and activity recognition using skin joints features and self-organizing map
18. Jalal, A., and Kim, S. (2005, January 1–4). The mechanism of edge detection using the block matching criteria for the motion estimation. Proceedings of the Conference on Human Computer Interaction, Daegu, Korea.
19. A Hybrid Feature Extraction Approach for Human Detection, Tracking and Activity Recognition Using Depth Sensors10.1007/s13369-015-1955-8 ↗
20. Azurdia-Meza, C.A., Falchetti, A., and Arrano, H.F. (2015, January 28–30). Evaluation of the improved parametric linear combination pulse in digital baseband communication systems. Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea.10.1109/ICTC.2015.7354591 ↗
21. Bongale, P., Ranjan, A., and Anand, S. (2012, January 25–27). Implementation of 3D object recognition and tracking. Proceedings of the 2012 International Conference on Recent Advances in Computing and Software Systems (RACSS), Chennai, India.10.1109/RACSS.2012.6212701 ↗
22. Depth Images-based Human Detection, Tracking and Activity Recognition Using Spatiotemporal Features and Modified HMM10.5370/JEET.2016.11.6.1857 ↗
23. Lai, K., Bo, L., Ren, X., and Fox, D. (2011, January 9–13). Sparse distance learning for object recognition combining RGB and depth information. Proceedings of the 2011 IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China.10.1109/ICRA.2011.5980377 ↗
24. Jalal, A., Kim, J.T., and Kim, T.-S. (2012, January 19). Development of a life logging system via depth imaging-based human activity recognition for smart homes. Proceedings of the International Symposium on Sustainable Healthy Buildings, Seoul, Korea.
25. Chang, J.-Y., Shyu, J.-J., and Cho, C.-W. (2009, January 8–10). Fuzzy rule inference based human activity recognition. Proceedings of the 2009 IEEE Control Applications, (CCA) & Intelligent Control, (ISIC), St. Petersburg, Russia.10.1109/CCA.2009.5280999 ↗
26. Human pose estimation and activity recognition from multi-view videos: Comparative explorations of recent developments10.1109/JSTSP.2012.2196975 ↗
27. LIBSVM: A library for support vector machines
28. A comprehensive survey of human action recognition with spatio-temporal interest point (STIP) detector10.1007/s00371-015-1066-2 ↗
29. Harris 3D: A robust extension of the Harris operator for interest point detection on 3D meshes10.1007/s00371-011-0610-y ↗
30. On space-time interest points10.1007/s11263-005-1838-7 ↗
31. Gilbert, A., Illingworth, J., and Bowden, R. (2008, January 12–18). Scale invariant action recognition using compound features mined from dense spatio-temporal corners. Proceedings of the European Conference on Computer Vision, Marseille, France.10.1007/978-3-540-88682-2_18 ↗
32. The recognition of human movement using temporal templates10.1109/34.910878 ↗
33. Hu, Y., Cao, L., Lv, F., Yan, S., and Gong, Y. (October, January 27). Action detection in complex scenes with spatial and temporal ambiguities. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.
34. View-independent human action recognition with volume motion template on single stereo camera10.1016/j.patrec.2009.11.017 ↗
35. Schuldt, C., Laptev, I., and Caputo, B. (2004, January 23–26). Recognizing human actions: A local SVM approach. Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, Cambridge, UK.10.1109/ICPR.2004.1334462 ↗
36. Sadanand, S., and Corso, J.J. (2012, January 16–21). Action bank: A high-level representation of activity in video. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA.10.1109/CVPR.2012.6247806 ↗
37. Wu, X., Xu, D., Duan, L., and Luo, J. (2011, January 20–25). Action recognition using context and appearance distribution features. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995624 ↗
38. Histogram of oriented rectangles: A new pose descriptor for human action recognition10.1016/j.imavis.2009.02.002 ↗
39. Peng, X., Qiao, Y., Peng, Q., and Qi, X. (2013, January 9–13). Exploring Motion Boundary based Sampling and Spatial-Temporal Context Descriptors for Action Recognition. Proceedings of the British Machine Vision Conference (BMVC), Bristol, UK.10.5244/C.27.59 ↗
40. Liu, J., Kuipers, B., and Savarese, S. (2011, January 20–25). Recognizing human actions by attributes. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995353 ↗
41. Action recognition using lie algebrized gaussians over dense local spatio-temporal features10.1007/s11042-013-1746-8 ↗
42. Wang, H., Kläser, A., and Schmid, C. (2011, January 20–25). Action recognition by dense trajectories. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995407 ↗
43. Rodriguez, M. (2010). Spatio-temporal Maximum Average Correlation Height Templates In Action Recognition And Video Summarization. [Ph.D. Thesis, University of Central Florida].
44. Soomro, K., and Zamir, A.R. (2014). Computer Vision in Sports, Springer.
45. Ma, S., Sigal, L., and Sclaroff, S. (2015, January 8–10). Space-time tree ensemble for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299137 ↗
46. Wang, C., Wang, Y., and Yuille, A.L. (2013, January 23–28). An approach to pose-based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.123 ↗
47. Kuehne, H., Jhuang, H., and Garrote, E. (2011, January 6–13). HMDB: A large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126543 ↗
48. Wang, H., and Schmid, C. (2013, January 3–6). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
49. Jiang, Y.-G., Dai, Q., Xue, X., Liu, W., and Ngo, C.W. (2012). European Conference on Computer Vision, Springer.
50. Kliper-Gross, O., Gurovich, Y., and Hassner, T. (2012). European Conference on Computer Vision, Springer.
51. Wang, L., Qiao, Y., and Tang, X. (2013, January 25–27). Motionlets: Mid-level 3D parts for human motion recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, Oregon.10.1109/CVPR.2013.345 ↗
52. Peng, X., Zou, C., Qiao, Y., and Peng, Q. (2014). European Conference on Computer Vision, Springer.
53. Jain, M., Jegou, H., and Bouthemy, P. (2013, January 23–28). Better exploiting motion for better action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.330 ↗
54. Fernando, B., Gavves, E., and Oramas, J.M. (2015, January 8–10). Modeling video evolution for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299176 ↗
55. Hoai, M., and Zisserman, A. (2014). Asian Conference on Computer Vision, Springer.
56. Marszalek, M., Laptev, I., and Schmid, C. (2009, January 20–26). Actions in context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206557 ↗
57. Vig, E., Dorr, M., and Cox, D. (2012). European Conference on Computer Vision, Springer.
58. Mathe, S., and Sminchisescu, C. (2012). Computer Vision–ECCV 2012, Springer.
59. Local polynomial space-time descriptors for action classification10.1007/s00138-014-0652-z ↗
60. Lan, T., Zhu, Y., Zamir, A.R., and Savarese, S. (2015, January 13–16). Action recognition by hierarchical mid-level action elements. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.517 ↗
61. Yuan, J., Liu, Z., and Wu, Y. (2009, January 20–26). Discriminative subvolume search for efficient action detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, Miami, FL, USA.
62. Action recognition using rate-invariant analysis of skeletal shape trajectories10.1109/TPAMI.2015.2439257 ↗
63. Zanfir, M., Leordeanu, M., and Sminchisescu, C. (2013, January 3–6). The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.342 ↗
64. Liu, J., Luo, J., and Shah, M. (2009, January 20–26). Recognizing realistic actions from videos “in the wild”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206744 ↗
65. Yilmaz, A., and Shah, M. (2005, January 20–26). Actions sketch: A novel action representation. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
66. Sheikh, Y., Sheikh, M., and Shah, M. (2005, January 17–20). Exploring the space of a human action. Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), Beijing, China.10.1109/ICCV.2005.90 ↗
67. Vision-based action recognition of construction workers using dense trajectories10.1016/j.aei.2016.04.009 ↗
68. Human Action Recognition in Unconstrained Videos by Explicit Motion Modeling10.1109/TIP.2015.2456412 ↗
69. Dollár, P., Rabaud, V., and Cottrell, G. (2005, January 15–16). Behavior recognition via sparse spatio-temporal features. Proceedings of the 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, China.
70. Thi, T.H., Zhang, J., Cheng, L., and Wang, L. (September, January 29). Human action recognition and localization in video using structured learning of local space-time features. Proceedings of the 2010 Seventh IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Boston, MA, USA.10.1109/AVSS.2010.76 ↗
71. Sivic, J., and Zisserman, A. (2003, January 14–17). Video Google: A text retrieval approach to object matching in videos. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238663 ↗
72. Bag of visual words and fusion methods for action recognition: Comprehensive study and good practice10.1016/j.cviu.2016.03.013 ↗
73. Liu, L., Wang, L., and Liu, X. (2011, January 6–13). In defense of soft-assignment coding. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.
74. Perronnin, F., Sánchez, J., and Mensink, T. (2010). European Conference on Computer Vision, Springer.
75. Dense trajectories and motion boundary descriptors for action recognition10.1007/s11263-012-0594-8 ↗
76. Li, H., and Greenspan, M. (2005, January 17–20). Multi-scale gesture recognition from time-varying contours. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV’05), Beijing, China.
77. Thurau, C., and Hlavác, V. (2008, January 24–26). Pose primitive based human action recognition in videos or still images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, Anchorage, AK, USA.10.1109/CVPR.2008.4587721 ↗
78. Efros, A.A., Berg, A.C., and Mori, G. (2003, January 14–17). Recognizing action at a distance. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238420 ↗
79. Fathi, A., and Mori, G. (2008, January 24–26). Action recognition by learning mid-level motion features. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, Anchorage, AK, USA.10.1109/CVPR.2008.4587735 ↗
80. Recognizing human actions by learning and matching shape-motion prototype trees10.1109/TPAMI.2011.147 ↗
81. Holte, M.B., Moeslund, T.B., and Nikolaidis, N. (2011, January 16–19). 3D human action recognition for multi-view camera systems. Proceedings of the 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), Hangzhou, China.10.1109/3DIMPVT.2011.50 ↗
82. Shape similarity for 3D video sequences of people10.1007/s11263-010-0319-9 ↗
83. Free viewpoint action recognition using motion history volumes10.1016/j.cviu.2006.07.013 ↗
84. Accurate 3D action recognition using learning on the Grassmann manifold10.1016/j.patcog.2014.08.011 ↗
85. Wang, L., and Suter, D. (2007, January 18–23). Recognizing human activities from silhouettes: Motion subspace and factorial discriminative graphical model. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383298 ↗
86. Recognising human actions by analysing negative spaces10.1049/iet-cvi.2011.0185 ↗
87. Hybrid classifier based human activity recognition using the silhouette and cells10.1016/j.eswa.2015.04.039 ↗
88. Silhouette-based human action recognition using SAX-Shapes10.1007/s00371-013-0842-0 ↗
89. Silhouette-based human action recognition using sequences of key poses10.1016/j.patrec.2013.01.021 ↗
90. A Low-Dimensional Radial Silhouette-Based Feature for Fast Human Action Recognition Fusing Multiple Views
91. Fast action recognition using negative space features10.1016/j.eswa.2013.07.082 ↗
92. Cheema, S., Eweiwi, A., and Thurau, C. (2011, January 6–13). Action recognition by learning discriminative key poses. Proceedings of the 2011 IEEE. International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130402 ↗
93. Human action recognition using histogram of motion intensity and direction from multiple views10.1049/iet-cvi.2015.0233 ↗
94. Multi-view Human Action Recognition using 2D Motion Templates based on MHIs and their HOG Description10.1049/iet-cvi.2015.0416 ↗
95. Ahmad, M., and Lee, S.-W. (2006, January 20–24). HMM-based human action recognition using multiview image sequences. Proceedings of the 18th International Conference on Pattern Recognition, ICPR 2006, Hong Kong, China.10.1109/ICPR.2006.630 ↗
96. A proposed unified framework for the recognition of human activity by exploiting the characteristics of action dynamics10.1016/j.robot.2015.11.013 ↗
97. Recognizing activities in multiple views with fusion of frame judgments10.1016/j.imavis.2014.01.006 ↗
98. Eweiwi, A., Cheema, S., and Thurau, C. (2011, January 6–13). Temporal key poses for human action recognition. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130403 ↗
99. Ojala, T., Pietikainen, M., and Harwood, D. (1994, January 9–13). Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. Proceedings of the 12th IAPR International Conference on Pattern Recognition, Jerusalem, Israel.
100. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns10.1109/TPAMI.2002.1017623 ↗