Improved Long Short-Term Memory Network with Multi-Attention for Human Action Flow Evaluation in Workshop
Source abstract
Abstract
As an indispensable part of workshops, the normalization of workers’ manufacturing processes is an important factor that affects product quality. How to effectively supervise the manufacturing process of workers has always been a difficult problem in intelligent manufacturing. This paper proposes a method for action detection and process evaluation of workers based on a deep learning model. In this method, the human skeleton and workpiece features are separately obtained by the monitoring frame and then input into an action detection network in chronological order. The model uses two inputs to predict frame-by-frame classification results, which are then merged into a continuous action flow, and finally, input into the action flow evaluation network. The network effectively improves the ability to evaluate action flow through the attention mechanism of key actions in the process. The experimental results show that our method can effectively recognize operation actions in workshops, and can evaluate the manufacturing process with 99% accuracy using the experimental verification dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Mechanical Engineering, Donghua University, Shanghai 201620, China
Shanghai Space Propulsion Technology Research Institute, Shanghai 201620, China
References (38)
Show source-provided references
1. Task Recognition from Joint Tracking Data in an Operational Manufacturing Cell10.1007/s10845-015-1168-8 ↗
2. Decision-Making Trends in Quality Management: A Literature Review about Industry 4.010.1590/0103-6513.20190086 ↗
3. Human Work and Status Evaluation Based on Wearable Sensors in Human Factors and Ergonomics: A Review10.1109/THMS.2018.2878824 ↗
4. Wang, D., Kotake, Y., Nakajima, H., Mori, K., and Hata, Y. (2018, January 7–10). A Relationship between Product Quality and Body Information of Worker and Its Application to Improvement of Productivity. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.10.1109/SMC.2018.00250 ↗
5. Simulation-Based Optimization Methodology for Offshore Natural Gas Liquefaction Process Design10.1021/ie403507p ↗
6. A New Threat Intelligence Scheme for Safeguarding Industry 4.0 Systems10.1109/ACCESS.2018.2844794 ↗
7. A Review on Human-Centered IoT-Connected Smart Labels for the Industry 4.010.1109/ACCESS.2018.2833501 ↗
8. Human Activity Recognition: A Survey10.1016/j.procs.2019.08.100 ↗
9. Kong, Y., and Fu, Y. (2018). Human Action Recognition and Prediction: A Survey. arXiv.
10. Lan, Z., Zhu, Y., Hauptmann, A.G., and Newsam, S. (2017, January 21–26). Deep Local Video Feature for Action Recognition. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.161 ↗
11. Dense Trajectories and Motion Boundary Descriptors for Action Recognition10.1007/s11263-012-0594-8 ↗
12. Wang, H., and Schmid, C. (2013, January 1–8). Action Recognition with Improved Trajectories. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
13. Wang, L., Qiao, Y., and Tang, X. (2015, January 7–12). Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299059 ↗
14. Human Action Recognition with 3D Convolution Skip-Connections and RNNs10.1007/978-3-030-04167-0_29 ↗
15. Tran, A., and Cheong, L.-F. (2017, January 22–29). Two-Stream Flow-Guided Convolutional Attention Networks for Action Recognition. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.368 ↗
16. Ke, Q., Bennamoun, M., An, S., Sohel, F., and Boussaid, F. (2017, January 21–26). A New Representation of Skeleton Sequences for 3D Action Recognition. Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.486 ↗
17. Human Activity Recognition Process Using 3-D Posture Data10.1109/THMS.2014.2377111 ↗
18. Wei, S., Song, Y., and Zhang, Y. (2017, January 17–20). Human Skeleton Tree Recurrent Neural Network with Joint Relative Motion Feature for Skeleton Based Action Recognition. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296249 ↗
19. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Dynamic Hand Gesture Recognition10.1186/s13640-019-0476-x ↗
20. Klochkov, Y., Gazizulina, A., Golovin, N., Glushkova, A., and Zh, S. (2017, January 18–20). Information Model-Based Forecasting of Technological Process State. Proceedings of the 2017 International Conference on Infocom Technologies and Unmanned Systems (Trends and Future Directions) (ICTUS), Dubai, UAE.10.1109/ICTUS.2017.8286099 ↗
21. A Human-in-the-Loop Manufacturing Control Architecture for the Next Generation of Production Systems10.1016/j.jmsy.2020.01.002 ↗
22. The Collaborative System Workflow Management of Industrial Design Based on Hierarchical Colored Petri-Net10.1109/ACCESS.2018.2809439 ↗
23. Zaremba, W., Sutskever, I., and Vinyals, O. (2014). Recurrent Neural Network Regularization. arXiv.
24. LSTM Recurrent Networks Learn Simple Context-Free and Context-Sensitive Languages10.1109/72.963769 ↗
25. Li, Y., Lan, C., Xing, J., Zeng, W., Yuan, C., and Liu, J. (2016, January 8–16). Online Human Action Detection Using Joint Classification-Regression Recurrent Neural Networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46478-7_13 ↗
26. Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection10.1109/TCSVT.2018.2799968 ↗
27. Shotton, J., FitzGibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. (2011, January 20–25). Real-Time Human Pose Recognition in Parts from Single Depth Images. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995316 ↗
28. Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21–26). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
29. Fully Convolutional Networks for Semantic Segmentation10.1109/TPAMI.2016.2572683 ↗
30. Welch, G., and Bishop, G. (1995). An Introduction to the Kalman Filter, University of North Carolina at Chapel Hill.
31. Fast Kalman-Like Optimal Unbiased FIR Filtering with Applications10.1109/TSP.2016.2516960 ↗
32. Sharma, S., Kiros, R., and Salakhutdinov, R. (2015). Action Recognition using Visual Attention. arXiv.
33. Bahdanau, D., Cho, K.H., and Bengio, Y. (2015, January 7–9). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA.
34. Uriarte-Arcia, A.V., López-Yáñez, I., and Yáñez-Márquez, C. (2014). One-Hot Vector Hybrid Associative Classifier for Medical Data Classification. PLoS ONE, 9.10.1371/journal.pone.0095715 ↗
35. Song, S., Lan, C., Xing, J., Zeng, W., and Liu, J. (2017, January 1–3). An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data. Proceedings of the 31st AAAI Conference on Artificial Intelligence, Marina del Rey, CA, USA.10.1609/aaai.v31i1.11212 ↗
36. Yuan, J., Liu, Z., and Wu, Y. (2009, January 20–25). Discriminative Subvolume Search for Efficient Action Detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), Miami, FL, USA.
37. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes from Videos in The Wild. arXiv.
38. Sáiz-Manzanares, M.C., Escolar-Llamazares, M.-C., and Arnaiz-González, Á (2020). Effectiveness of Blended Learning in Nursing Education. Int. J. Environ. Res. Public Health, 17.10.3390/ijerph17051589 ↗