Sensor-Based Hand Gesture Detection and Recognition by Key Intervals
Source abstract
Abstract
This study aims to present a novel neural network architecture for sensor-based gesture detection and recognition. The algorithm is able to detect and classify accurately a sequence of hand gestures from the sensory data produced by accelerometers and gyroscopes. Each hand gesture in the sequence is regarded as an object with a pair of key intervals. The detection and classification of each gesture are equivalent to the identification and matching of the corresponding key intervals. A simple automatic labelling is proposed for the identification of key intervals without manual inspection of sensory data. This could facilitate the collection and annotation of training data. To attain superior generalization and regularization, a multitask learning algorithm for the simultaneous training for gesture detection and classification is proposed. A prototype system based on smart phones for remote control of home appliances was implemented for the performance evaluation. Experimental results reveal that the proposed algorithm provides an effective alternative for applications where accurate detection and classification of hand gestures by simple networks are desired.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · IoT-based Smart Home Systems · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei 116, Taiwan
NVIDIA AI Technology Center, Taipei 114, Taiwan
Funding
Ministry of Science and Technology, Taiwan
References (35)
Show source-provided references
1. Online Dynamic Hand Gesture Recognition Including Energy Analysis10.1109/TBIOM.2020.2968216 ↗
2. Deep Learning for Sensor-Based Human Activity Recognition: Overview, Challenged, and Opportunities
3. Tan, T.-H., Badarch, L., Zeng, W.-X., Gochoo, M., Alnajjar, F.S., and Hsieh, J.-W. (2021). Binary Sensors-Based Privacy-Preserved Activity Recognition of Elderly Living Alone Using an RNN. Sensors, 21.10.3390/s21165371 ↗
4. A Continuous Hand Gestures Recognition Technique for Human–Machine Interaction Using Accelerometer and Gyroscope Sens10.1109/JSEN.2016.2581023 ↗
5. Similarity Matching-Based Extensible Hand Gesture Recognition10.1109/JSEN.2015.2392091 ↗
6. Wu, J., Pan, G., Zhang, D., Qi, G., and Li, S. (2009, January 7–9). Gesture Recognition with a 3-D Accelerometer. Proceedings of the International Conference on Ubiquitous Intelligence and Computing, Brisbane, QLD, Australia.10.1007/978-3-642-02830-4_4 ↗
7. Sensor-Based Continuous Hand Gesture Recognition by Long Short-Term Memory10.1109/LSENS.2018.2864963 ↗
8. Zhao, T., Liu, J., Wang, Y., Liu, H., and Chen, Y. (2018, January 15–19). PPG-Based Finger-Level Gesture Recognition Leveraging Wearables. Proceedings of the INFOCOM 2018-IEEE Conference on Computer Communications, Honolulu, HI, USA.10.1109/INFOCOM.2018.8486006 ↗
9. Pathak, V., Mongia, S., and Chitranshi, G. (2015, January 21–24). A Framework for Hand Gesture Recognition Based on Fusion of Flex, Contact and Accelerometer Sensor. Proceedings of the Conference on Image Information Processing, Waknaghat, India.10.1109/ICIIP.2015.7414787 ↗
10. A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sens10.1109/TSMCA.2011.2116004 ↗
11. uWave: Accelerometer-based personalized gesture recognition and its applications10.1016/j.pmcj.2009.07.007 ↗
12. Ducloux, J., Colla, P., Petrashin, P., Lancioni, W., and Toledo, L. (2014, January 12–15). Accelerometer-Based Hand Gesture Recognition System for Interaction in Digital TV. Proceedings of the International Instrumentation and Measurement Technology Conference (I2MTC) Proceedings, Montevideo, Uruguay.10.1109/I2MTC.2014.6861003 ↗
13. Lee, D., Yoon, H., and Kim, J. (2016, January 16–19). Continuous gesture recognition by using gesture spotting. Proceedings of the 16th International Conference on Control, Automation and Systems (ICCAS), Gyeongju, Korea.10.1109/ICCAS.2016.7832502 ↗
14. Wearable Sensor-Based Hand Gesture and Daily Activity Recognition for Robot-Assisted Living10.1109/TSMCA.2010.2093883 ↗
15. Agrawal, S., Constandache, I., Gaonkar, S., Choudhury, R.R., Caves, K., and DeRuyter, F. (July, January 28). Using Mobile Phones to Write in Air. Proceedings of the 9th International Conference on Mobile Systems, Applications, and Servicess, Bethesda, MD, USA.
16. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
17. Inertial Gesture Recognition with BLSTM-RNN10.1007/978-3-319-09903-3_19 ↗
18. Lee, S.M., Yoon, S.M., and Cho, H. (2017, January 13–16). Human Activity Recognition From Accelerometer Data Using Convolutional Neural Network. Proceedings of the IEEE International Conference on Big Data and Smart Computing, Jeju, Korea.
19. Chu, Y.C., Jhang, Y.J., Tai, T.M., and Hwang, W.J. (2020). Recognition of Hand Gesture Sequences by Accelerometers and Gyroscope. Appl. Sci., 10.10.3390/app10186507 ↗
20. Multitask Learning10.1023/A:1007379606734 ↗
21. A Survey of Deep Learning Based Object Detection10.1109/ACCESS.2019.2939201 ↗
22. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards Real-Time Object Detection with Region Networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
23. Fu, C.Y., Liu, W., Ranga, A., Tyagi, A., and Berg, A.C. (2017). DSSD: Deconvolutional Single Shot Detector. arXiv.
24. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017). Focal Loss for Dense Object Detection. arXiv.10.1109/ICCV.2017.324 ↗
25. Zhou, X., Wang, D., and Krahenbuhl, P. (2019). Objects as Points. arXiv.
26. Law, H., and Deng, J. (2018, January 8–14). CornerNet: Detecting Objects as Paired Keypoints. Proceedings of the European Conference Computer Vision, Munich, Germany.10.1007/978-3-030-01264-9_45 ↗
27. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., and Tian, Q. (2019). CenterNet: Keypoint Triplets for Object Detection. arXiv.10.1109/ICCV.2019.00667 ↗
28. Zhou, T., Wang, W., Liu, S., Yang, Y., and Gool, L.V. (2021, January 20–25). Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00167 ↗
29. Yang, L., Song, Q., Wang, Z., and Jiang, M. (2019, January 15–20). Parsing R-CNN for Instance-Level Human Analysis. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00045 ↗
30. Li, J., Zhao, J., Wei, Y., Lang, C., Li, Y., Sim, T., Yan, S., and Feng, J. (2017). Multiple-Human Parsing in the Wild. arXiv.
31. Wu, Y., and He, K. (2018, January 8–14). Group Normalization. Proceedings of the European Conference Computer Vision, Munich, Germany.10.1007/978-3-030-01261-8_1 ↗
32. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
33. Chollet, F. (2022, March 30). Keras. Available online: http://github.com/fchollet/keras.
34. An introduction to ROC analysis10.1016/j.patrec.2005.10.010 ↗
35. A guided tour to approximate string matching10.1145/375360.375365 ↗