journal article·2022-09-27Open access

LSTM Recurrent Neural Network for Hand Gesture Recognition Using EMG Signals

Alejandro Toro-Ossaba, Juan Jaramillo-Tigreros, Juan C. Tejada, Alejandro Peña, Alexandro López-González, Rui Alexandre Castanho

Applied Sciences· Volume 12· Issue 19

Abstract

Currently, research on gesture recognition systems has been on the rise due to the capabilities these systems provide to the field of human–machine interaction, however, gesture recognition in prosthesis and orthesis has been carried out through the use of an extensive amount of channels and electrodes to acquire the EMG (Electromyography) signals, increasing the cost and complexity of these systems. The scientific literature shows different approaches related to gesture recognition based on the analysis of EMG signals using deep learning models, highlighting the recurrent neural networks with deep learning structures. This paper presents the implementation of a Recurrent Neural Network (RNN) model using Long-short Term Memory (LSTM) units and dense layers to develop a gesture classifier for hand prosthesis control, aiming to decrease the number of EMG channels and the overall model complexity, in order to increase its scalability for embedded systems. The proposed model requires the use of only four EMG channels to recognize five hand gestures, greatly reducing the number of electrodes compared to other approaches found in the literature. The proposed model was trained using a dataset for each gesture EMG signals, which were recorded for 20 s using a custom EMG armband. The model reached an accuracy of to 99% for the training and validation stages, and an accuracy of 87 ± 7% during real-time testing. The results obtained by the proposed model establish a general methodology for the reduction of complexity in the recognition of gestures intended for human.machine interaction for different computational devices.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Muscle activation and electromyography studies · EEG and Brain-Computer Interfaces · Hand Gesture Recognition Systems

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Grupo de Investigación en Inteligencia Computacional y Automática (GIICA), Universidad EIA, Envigado 055428, Colombia

Departamento de Estudios en Ingeniería para la Innovación, Universidad Iberoamericana, Ciudad de México 01219, Mexico

Grupo de Investigación en Información y Gestión, Escuela de Administración, Universidad EAFIT, Medellín 050021, Colombia

Faculty of Applied Sciences, WSB University, 03-204 Dabrowa Gornicza, Poland

College of Business and Economics, University of Johannesburg, Auckland Park, P.O. Box 524, Johannesburg 2006, South Africa

Funding

Minister of Education and Science · Universidad EIA · Universidad Iberoamericana

References (79)

Show source-provided references
1. Hand Gesture Recognition for Human Computer Interaction10.1016/j.procs.2017.09.092 ↗
2. Hand gestures recognition from surface electromyogram signal based on self-organizing mapping and radial basis function network10.1016/j.bspc.2021.102629 ↗
3. A bionic hand controlled by hand gesture recognition based on surface EMG signals: A preliminary study10.1016/j.bbe.2017.11.001 ↗
4. Recent methods and databases in vision-based hand gesture recognition: A review10.1016/j.cviu.2015.08.004 ↗
5. Arunraj, M., Srinivasan, A., and Arjunan, S.P. (2020). A Real-Time Capable Linear Time Classifier Scheme for Anticipated Hand Movements Recognition from Amputee Subjects Using Surface EMG Signals. IRBM.10.1016/j.irbm.2020.08.003 ↗
6. Spectral Collaborative Representation based Classification for hand gestures recognition on electromyography signals10.1016/j.bspc.2015.09.001 ↗
7. Online myoelectric pattern recognition based on hybrid spatial features10.1016/j.bspc.2021.102482 ↗
8. Netlab MLP - Performance Evaluation for Pattern Recognition in Myoletric Signal10.1016/j.procs.2018.04.092 ↗
9. Automatic hand gesture recognition using hybrid meta-heuristic-based feature selection and classification with Dynamic Time Warping10.1016/j.cosrev.2020.100320 ↗
10. The effect of arm position on classification of hand gestures with intramuscular EMG10.1016/j.bspc.2018.02.013 ↗
11. Medical gesture recognition using dynamic arc length warping10.1016/j.bspc.2019.04.022 ↗
12. User-adaptive hand gesture recognition system with interactive training10.1016/j.imavis.2005.07.016 ↗
13. EMG-based online classification of gestures with recurrent neural networks10.1016/j.patrec.2019.07.021 ↗
14. A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition10.1016/j.cviu.2008.12.001 ↗
15. Systolic hand gesture recognition/detection system based on FPGA with multi-port BRAMs10.1016/j.aej.2019.05.018 ↗
16. Deep attention network for joint hand gesture localization and recognition using static RGB-D images10.1016/j.ins.2018.02.024 ↗
17. A novel autonomous learning framework to enhance sEMG-based hand gesture recognition using depth information10.1016/j.bspc.2021.102444 ↗
18. A multi-scale descriptor for real time RGB-D hand gesture recognition10.1016/j.patrec.2020.11.011 ↗
19. A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion10.1016/j.entcom.2020.100373 ↗
20. Percentage estimation of muscular activity of the forearm by means of EMG signals based on the gesture recognized using CNN10.1016/j.sbsr.2020.100353 ↗
21. Hand gesture recognition based on dynamic Bayesian network framework10.1016/j.patcog.2010.03.016 ↗
22. A novel shape matching descriptor for real-time hand gesture recognition10.1016/j.cviu.2021.103241 ↗
23. Multimodal hand gesture recognition combining temporal and pose information based on CNN descriptors and histogram of cumulative magnitudes10.1016/j.jvcir.2020.102772 ↗
24. Intelligent human computer interaction based on non redundant EMG signal10.1016/j.aej.2020.01.015 ↗
25. EMG and ENG-envelope pattern recognition for prosthetic hand control10.1016/j.jneumeth.2018.10.004 ↗
26. Hand movement recognition from sEMG signals using Fourier decomposition method10.1016/j.bbe.2021.03.004 ↗
27. Recurrent Neural Network for electromyographic gesture recognition in transhumeral amputees10.1016/j.asoc.2020.106616 ↗
28. EMG map image processing for recognition of fingers movement10.1016/j.jelekin.2019.102364 ↗
29. Centroid tracking based dynamic hand gesture recognition using discrete Hidden Markov Models10.1016/j.neucom.2016.06.075 ↗
30. Hand gesture recognition based on motor unit spike trains decoded from high-density electromyography10.1016/j.bspc.2019.101637 ↗
31. A novel, co-located EMG-FMG-sensing wearable armband for hand gesture recognition10.1016/j.sna.2019.111738 ↗
32. Exploration of Force Myography and surface Electromyography in hand gesture classification10.1016/j.medengphy.2017.01.015 ↗
33. A novel CNN-LSTM-based approach to predict urban expansion10.1016/j.ecoinf.2021.101325 ↗
34. Saleh, N.M., Zaini, N., and Latip, M.F.A. (2019, January 7). Recurrent neural networks-gated recurrent unit for behaviour analysis and prediction modelling to trigger high energy-consumption alert. Proceedings of the 2019 IEEE 9th International Conference on System Engineering andvTechnology (ICSET), Shah Alam, Malaysia.10.1109/ICSEngT.2019.8906493 ↗
35. Jabbari, M., Khushaba, R.N., and Nazarpour, K. (2020, January 20–24). EMG-Based Hand Gesture Classification with Long Short-Term Memory Deep Recurrent Neural Networks. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), Montreal, QC, Canada.10.1109/EMBC44109.2020.9175279 ↗
36. Samadani, A. (2018, January 18–21). Gated Recurrent Neural Networks for EMG-Based Hand Gesture Classification. A Comparative Study. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), Honolulu, HI, USA.10.1109/EMBC.2018.8512531 ↗
37. Geng, W., Hu, Y., Wong, Y., Wei, W., Du, Y., and Kankanhalli, M. (2018). A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition. PLoS ONE, 13.10.1371/journal.pone.0206049 ↗
38. Zhang, Z., He, C., and Yang, K. (2020). A Novel Surface Electromyographic Signal-Based Hand Gesture Prediction Using a Recurrent Neural Network. Sensors, 20.10.3390/s20143994 ↗
39. Jiang, Y., Song, L., Zhang, J., Song, Y., and Yan, M. (2022). Multi-Category Gesture Recognition Modeling Based on sEMG and IMU Signals. Sensors, 22.10.3390/s22155855 ↗
40. A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration10.1016/j.bspc.2019.02.011 ↗
41. Xie, B., Li, B., and Harland, A. (2018). Movement and Gesture Recognition Using Deep Learning and Wearable-sensor Technology. ACM Int. Conf. Proc. Ser., 26–31.10.1145/3268866.3268890 ↗
42. He, Y., Fukuda, O., Bu, N., Okumura, H., and Yamaguchi, N. (2018, January 18–21). Surface EMG Pattern Recognition Using Long Short-Term Memory Combined with Multilayer Perceptron. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI, USA.10.1109/EMBC.2018.8513595 ↗
43. Nordin, M., and Frankel, V.H. (2012). Basic Biomechanics of the Musculoskeletal System, Lippincott Williams & Wilkins. [4th ed.].
44. Toro Ossaba, A., Jaramillo Tigreros, J.J., and Tejada Orjuela, J.C. (2020, January 4–6). Open Source Multichannel EMG Armband design. Proceedings of the 2020 IX International Congress of Mechatronics Engineering and Automation (CIIMA), Cartagena, Colombia.10.1109/CIIMA50553.2020.9290291 ↗
45. Asogbon, M.G., Samuel, O.W., Jiang, Y., Wang, L., Geng, Y., Sangaiah, A.K., Chen, S., Fang, P., and Li, G. (2020). Appropriate Feature Set and Window Parameters Selection for Efficient Motion Intent Characterization towards Intelligently Smart EMG-PR System. Symmetry, 12.10.3390/sym12101710 ↗
46. Merletti, R., and Farina, D. (2016). Surface Electromyography: Physiology, Engineering and Applications, Wiley.10.1002/9781119082934 ↗
47. Objectives, criteria and methods for the design of the SmartHand transradial prosthesis10.1017/S0263574709990750 ↗
48. Ventimiglia, P., Padir, T., and Schaufeld, J. (2012). Design of a Human Hand Prosthesis. A Major Qualifying Project Report. [Bachelor’s Thesis, Faculty of the Worcester Polytechnic Institute].
49. Mechanical design and performance specifications of anthropomorphic prosthetic hands: A review10.1682/JRRD.2011.10.0188 ↗
50. Literature review on needs of upper limb prosthesis users10.3389/fnins.2016.00209 ↗
51. Determining the optimal window length for pattern recognition-based myoelectric control: Balancing the competing effects of classification error and controller delay10.1109/TNSRE.2010.2100828 ↗
52. Feature reduction and selection for EMG signal classification10.1016/j.eswa.2012.01.102 ↗
53. Moin, A., Zhou, A., Rahimi, A., Benatti, S., Menon, A., Tamakloe, S., Ting, J., Yamamoto, N., Khan, Y., and Burghardt, F. (2018, January 27–30). An EMG Gesture Recognition System with Flexible High-Density Sensors and Brain-Inspired High-Dimensional Classifier. Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy.10.1109/ISCAS.2018.8351613 ↗
54. Jaramillo-Yánez, A., Benalcázar, M.E., and Mena-Maldonado, E. (2020). Real-time hand gesture recognition using surface electromyography and machine learning: A systematic literature review. Sensors, 20.10.3390/s20092467 ↗
55. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
56. Smith, S.W. (1999). The Scientist and Engineer’s Guide to Digital Signal Processing, California Technical Publishing. [2nd ed.].
57. A review of irregular time series data handling with gated recurrent neural networks10.1016/j.neucom.2021.02.046 ↗
58. Wang, Y., Perry, M., Whitlock, D., and Sutherland, J.W. (2020). Detecting anomalies in time series data from a manufacturing system using recurrent neural networks. J. Manuf. Syst.
59. A review of recurrent neural networks: LSTM cells and network architectures10.1162/neco_a_01199 ↗
60. Recurrent Neural Networks for Time Series Forecasting: Current status and future directions10.1016/j.ijforecast.2020.06.008 ↗
61. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv.
62. Józefowicz, R., Zaremba, W., and Sutskever, I. (2015, January 6–11). An Empirical Exploration of Recurrent Network Architectures. Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), Lille, France.
63. Mangal, S., Joshi, P., and Modak, R. (2019). LSTM vs. GRU vs. Bidirectional RNN for script generation. arXiv.
64. Yang, S., Yu, X., and Zhou, Y. (2020, January 12–14). LSTM and GRU Neural Network Performance Comparison Study: Taking Yelp Review Dataset as an Example. Proceedings of the 2020 International Workshop on Electronic Communication and Artificial Intelligence (IWECAI), Shanghai, China.10.1109/IWECAI50956.2020.00027 ↗
65. Mohammed, A.A., and Umaashankar, V. (2018, January 19–22). Effectiveness of Hierarchical Softmax in Large Scale Classification Tasks. Proceedings of the 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Bangalore, India.10.1109/ICACCI.2018.8554637 ↗
66. Activation Function in Neural Networks
67. Kingma, D.P., and Ba, J.L. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA.
68. Adam Optimization Algorithm for Wide and Deep Neural Network10.17977/um018v2i12019p41-46 ↗
69. Sakinah, N., Tahir, M., Badriyah, T., and Syarif, I. (2019, January 27–28). LSTM with Adam Optimization-Powered High Accuracy Preeclampsia Classification. Proceedings of the IES 2019—International Electronics Symposium: The Role of Techno-Intelligence in Creating an Open Energy System Towards Energy Democracy, Surabaya, Indonesia.10.1109/ELECSYM.2019.8901536 ↗
70. Défossez, A., Bottou, L., Bach, F., and Usunier, N. (2020). A Simple Convergence Proof of Adam and Adagrad. arXiv.
71. Feurer, M., and Hutter, F. (2019). Automated Machine Learning, Springer.
72. Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization
73. HELP: An LSTM-based approach to hyperparameter exploration in neural network learning10.1016/j.neucom.2020.12.133 ↗
74. Random Search for Hyper-Parameter Optimization
75. Two-stage optimization for machine learning workflow10.1016/j.is.2019.101483 ↗
76. Grandini, M., Bagli, E., and Visani, G. (2020). Metrics for Multi-Class Classification: An Overview. arXiv.
77. Igual, C., Pardo, L.A., Hahne, J.M., and Igual, J. (2019). Myoelectric Control for Upper Limb Prostheses. Electronics, 8.10.3390/electronics8111244 ↗
78. Trends and Challenges in EMG Based Control Scheme of Exoskeleton Robots—A Review
79. The role of muscle synergies in myoelectric control: Trends and challenges for simultaneous multifunction control10.1088/1741-2560/11/5/051001 ↗