Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos
Source abstract
Abstract
Video classification is an essential process for analyzing the pervasive semantic information of video content in computer vision. Traditional hand-crafted features are insufficient when classifying complex video information due to the similarity of visual contents with different illumination conditions. Prior studies of video classifications focused on the relationship between the standalone streams themselves. In this paper, by leveraging the effects of deep learning methodologies, we propose a two-stream neural network concept, named state-exchanging long short-term memory (SE-LSTM). With the model of spatial motion state-exchanging, the SE-LSTM can classify dynamic patterns of videos using appearance and motion features. The SE-LSTM extends the general purpose of LSTM by exchanging the information with previous cell states of both appearance and motion stream. We propose a novel two-stream model Dual-CNNSELSTM utilizing the SE-LSTM concept combined with a Convolutional Neural Network, and use various video datasets to validate the proposed architecture. The experimental results demonstrate that the performance of the proposed two-stream Dual-CNNSELSTM architecture significantly outperforms other datasets, achieving accuracies of 81.62%, 79.87%, and 69.86% with hand gestures, fireworks displays, and HMDB51 datasets, respectively. Furthermore, the overall results signify that the proposed model is most suited to static background dynamic patterns classifications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Information Engineering, National Central University, Taoyuan 32001, Taiwan
Funding
Ministry of Science and Technology, Taiwan
References (52)
Show source-provided references
1. Learning Hierarchical Features for Scene Labeling10.1109/TPAMI.2012.231 ↗
2. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2013). Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv.10.1109/CVPR.2014.81 ↗
3. Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
4. 3D Convolutional Neural Networks for Human Action Recognition10.1109/TPAMI.2012.59 ↗
5. Learning long-term dependencies with gradient descent is difficult10.1109/72.279181 ↗
6. Two-Stream Convolutional Networks for Action Recognition in Videos
7. Wang, H., and Schmid, C. (2013, January 1–8). Action Recognition with Improved Trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
8. Ng, J.Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015). Beyond Short Snippets: Deep Networks for Video Classification. arXiv.
9. Wu, Z., Wang, X., Jiang, Y.-G., Ye, H., and Xue, X. (2015). Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification. arXiv.10.1145/2733373.2806222 ↗
10. Shahroudy, A., Liu, J., Ng, T.-T., and Wang, G. (2016, January 27–30). Ntu rgb+ d: A large scale dataset for 3d human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.115 ↗
11. Singh, B., Marks, T.K., Jones, M., Tuzel, O., and Shao, M. (2016, January 27–30). A multi-stream bi-directional recurrent neural network for fine-grained action detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.216 ↗
12. Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video10.1007/s11263-016-0957-7 ↗
13. Du, Y., Wang, W., and Wang, L. (2015, January 7–12). Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
14. Veeriah, V., Zhuang, N., and Qi, G.-J. (2015, January 13–16). Differential recurrent neural networks for action recognition. Proceedings of the IEEE International Conference on Computer Vision, Convention Center, Santiago, Chile.10.1109/ICCV.2015.460 ↗
15. Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., and Kautz, J. (2016, January 27–30). Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.456 ↗
16. Hakim, N.L., Shih, T.K., Kasthuri Arachchi, S.P., Aditya, W., Chen, Y.-C., and Lin, C.-Y. (2019). Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model. Sensors, 19.10.3390/s19245429 ↗
17. Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., and Vijayanarasimhan, S. (2016). YouTube-8M: A Large-Scale Video Classification Benchmark. arXiv.
18. Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks10.1109/TPAMI.2017.2670560 ↗
19. Deep Learning-based Firework Video Pattern Classification
20. (2020, January 19). Available online: http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/.
21. On space-time interest points10.1007/s11263-005-1838-7 ↗
22. Laptev, M., Marszalek Schmid, C., and Rozenfeld, B. (2008, January 23–28). Learning Realistic Human Actions from Movies. Proceedings of the CVPR, Anchorage, AK, USA.10.1109/CVPR.2008.4587756 ↗
23. Wang, H., Ullah, M.M., Klaser, A., Laptev, I., and Schmid, C. (2009, January 7–10). Evaluation of Local Spatio-Temporal Features for Action Recognition. Proceedings of the British Machine Vision Conference (BMVC), London, UK.10.5244/C.23.124 ↗
24. Oneata, D., Verbeek, J., and Schmid, C. (2013, January 1–8). Action and event recognition with fisher vectors on a compact feature set. Proceedings of the ICCV, Sydney, NSW, Australia.10.1109/ICCV.2013.228 ↗
25. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 8–10). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
26. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 24–27). Large-scale Video Classification with Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
27. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE international conference on computer vision, Washington, DC, USA.10.1109/ICCV.2015.510 ↗
28. Wang, L., Qiao, Y., and Tang, X. (2015, January 8–10). Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299059 ↗
29. Tang, K., Fei-Fei, L., and Koller, D. (2012, January 18–20). Learning latent temporal structure for complex event detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247808 ↗
30. Wang, Y., and Mori, G. (2009, January 20–25). Max-margin hidden conditional random fields for human action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206709 ↗
31. Fernando, B., Gavves, E., Oramas, J.M., Ghodrati, A., and Tuytelaars, T. (2015, January 8–10). Modeling Video Evolution for Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299176 ↗
32. Donahue, J., Hendricks, L.A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
33. Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., and Courville, A. (2015). Describing Videos by Exploiting Temporal Structure. arXiv.10.1109/ICCV.2015.512 ↗
34. Zhang, H., Wang, M., Hong, R., and Chua, T.S. (2016, January 15–19). Play and Rewind: Optimizing Binary Representations of Videos by Self-Supervised Temporal Hashing. Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2964308 ↗
35. Graves, A., Mohamed, A., and Hinton, G. (2013). Speech Recognition with Deep Recurrent Neural Networks. arXiv.10.1109/ICASSP.2013.6638947 ↗
36. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
37. A survey of glove-based input10.1109/38.250916 ↗
38. Real-time hand-tracking with a color glove
39. Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove
40. Deep learning in vision-based static hand gesture recognition10.1007/s00521-016-2294-8 ↗
41. Molchanov, P., Gupta, S., Kim, K., and Kautz, J. (2015, January 7–12). Hand gesture recognition with 3D convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA.10.1109/CVPRW.2015.7301342 ↗
42. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–8). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS 2012), Lake Tahoe, NV, USA.
43. Pigou, L., Dieleman, S., Kindermans, P.-J., and Schrauwen, B. (2014). Sign Language Recognition Using Convolutional Neural Networks, Springer.
44. (2019, April 26). Home -Keras Documentation. Available online: https://keras.io/.
45. (2019, April 26). Understanding LSTM Networks -Colah’s blog. Available online: http://colah.github.io/posts/2015-08-Understanding-LSTMs/.
46. (2020, February 18). Deep Learning Based Dynamic Patterns Classification of Videos Using Spatio-Temporal Virtual Communication. Available online: https://srilankanstudytaiwan.wixsite.com/twostreamvclstm/.
47. Khong, V.M., and Tran, T.H. (2018, January 5–6). Improving human action recognition with two-stream 3d convolutional neural network. Proceedings of the 2018 1st International Conference on Multimedia Analysis and Pattern Recognition (MAPR), Ho Chi Minh City, Vietnam.10.1109/MAPR.2018.8337518 ↗
48. Farnebäck, G. (2003). Two-Frame Motion Estimation Based on Polynomial Expansion. Scandinavian Conference on Image Analysis, Halmstad, Sweden, 29 June–2 July 2003, Springer.10.1007/3-540-45103-X_50 ↗
49. Kingma, D.P., and Adam Ba, J. (2015, January 7–9). A Method for Stochastic Optimization. Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA, USA.
50. Igel, C., and Hüsken, M. (2000, January 23–26). Improving the Rprop Learning Algorithm. Proceedings of the Second International Symposium on Neural Computation, NC’2000, ICSC, Berlin, Germany.
51. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27–30). Convolutional Two-Stream Network Fusion for Video Action Recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.213 ↗
52. Feichtenhofer, C., Pinz, A., and Wildes, R.P. (2016, January 5–10). Spatiotemporal Residual Networks for Video Action Recognition. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.10.1109/CVPR.2017.787 ↗