Training-Free Acoustic-Based Hand Gesture Tracking on Smart Speakers
Source abstract
Abstract
Hand gesture recognition is an essential Human–Computer Interaction (HCI) mechanism for users to control smart devices. While traditional device-based methods support acceptable recognition performance, the recent advance in wireless sensing could enable device-free hand gesture recognition. However, two severe limitations are serious environmental interference and high-cost hardware, which hamper wide deployment. This paper proposes the novel system TaGesture, which employs an inaudible acoustic signal to realize device-free and training-free hand gesture recognition with a commercial speaker and microphone array. We address unique technical challenges, such as proposing a novel acoustic hand-tracking-smoothing algorithm with an Interaction Multiple Model (IMM) Kalman Filter to address the issue of localization angle ambiguity, and designing a classification algorithm to realize acoustic-based hand gesture recognition without training. Comprehensive experiments are conducted to evaluate TaGesture. Results show that it can achieve a total accuracy of 97.5% for acoustic-based hand gesture recognition, and support the furthest sensing range of up to 3 m.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Indoor and Outdoor Localization Technologies · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China
Funding
China Postdoctoral Science Foundation
References (26)
Show source-provided references
1. Wang, C., Liu, J., Chen, Y., Liu, H., Xie, L., Wang, W., He, B., and Lu, S. (2018, January 15–19). Multi-touch in the air: Device-free finger tracking and gesture recognition via cots rfid. Proceedings of the IEEE INFOCOM 2018—IEEE Conference on Computer Communications, Honolulu, HI, USA.10.1109/INFOCOM.2018.8486346 ↗
2. A hierarchical hand gesture recognition framework for sports referee training-based EMG and accelerometer sensors10.1109/TCYB.2020.3007173 ↗
3. Han, H., and Yoon, S.W. (2019). Gyroscope-based continuous human hand gesture recognition for multi-modal wearable input device for human machine interaction. Sensors, 19.10.3390/s19112562 ↗
4. Koch, P., Dreier, M., Böhme, M., Maass, M., Phan, H., and Mertins, A. (2019, January 2–6). Inhomogeneously stacked rnn for recognizing hand gestures from magnetometer data. Proceedings of the 2019 27th European Signal Processing Conference (EUSIPCO), A Coruña, Spain.10.23919/EUSIPCO.2019.8903132 ↗
5. Video hand gestures recognition using depth camera and lightweight cnn10.1109/JSEN.2022.3181518 ↗
6. Li, C., Liu, M., and Cao, Z. (2020, January 6–9). WiHF: Enable user identified gesture recognition with WiFi. Proceedings of the IEEE INFOCOM 2020—IEEE Conference on Computer Communications, Toronto, ON, Canada.10.1109/INFOCOM41043.2020.9155539 ↗
7. Xie, B., and Xiong, J. (2020, January 16–19). Combating interference for long range LoRa sensing. Proceedings of the 18th Conference on Embedded Networked Sensor Systems, Virtual.10.1145/3384419.3430731 ↗
8. Gesture recognition using ambient light10.1145/3191772 ↗
9. Radar-Based Air-Writing Gesture Recognition Using a Novel Multistream CNN Approach10.1109/JIOT.2022.3189395 ↗
10. Ultragesture: Fine-grained gesture sensing and recognition
11. Li, D., Liu, J., Lee, S.I., and Xiong, J. (2020, January 16–19). FM-track: Pushing the limits of contactless multi-target tracking using acoustic signals. Proceedings of the 18th Conference on Embedded Networked Sensor Systems, Virtual.10.1145/3384419.3430780 ↗
12. Hearing at low and infrasonic frequencies
13. Ubiquitous acoustic sensing on commodity iot devices: A survey10.1109/COMST.2022.3145856 ↗
14. Li, D., Liu, J., Lee, S.I., and Xiong, J. (2023, January 6–9). Room-Scale Hand Gesture Recognition Using Smart Speakers. Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, Boston, MA, USA.10.1145/3560905.3568528 ↗
15. Towards robust gesture recognition by characterizing the sensing quality of WiFi signals
16. Sign language/gesture recognition based on cumulative distribution density features using UWB radar
17. Webber, J., and Mehbodniya, A. (2022, January 6–8). Recognition of Hand Gestures using Visible Light and a Probabilistic-based Neural Network. Proceedings of the 2022 4th IEEE Middle East and North Africa COMMunications Conference (MENACOMM), Amman, Jordan.10.1109/MENACOMM57252.2022.9998225 ↗
18. Yin, Y., Yu, X., Gao, S., Yang, X., Chen, P., and Niu, Q. (2022, January 24–26). MineSOS: Long-Range LoRa-Based Distress Gesture Sensing for Coal Mine Rescue. Proceedings of the International Conference on Wireless Algorithms, Systems, and Applications, Dalian, China.10.1007/978-3-031-19214-2_9 ↗
19. Edge machine learning techniques applied to rfid for device-free hand gesture recognition10.1109/JRFID.2022.3185804 ↗
20. Lian, J., Lou, J., Chen, L., and Yuan, X. (2021). Echospot: Spotting your locations via acoustic sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 5.10.1145/3478095 ↗
21. Push the limit of acoustic gesture recognition10.1109/TMC.2020.3032278 ↗
22. Wang, P., Jiang, R., Guo, Z., and Liu, C. (2023). Afitness: Fitness Monitoring on Smart Devices via Acoustic Motion Images. ACM Trans. Sens. Netw.10.1145/3592612 ↗
23. EchoSensor: Fine-Grained Ultrasonic Sensing for Smart Home Intrusion Detection10.1145/3615658 ↗
24. Lian, J., Yuan, X., Li, M., and Tzeng, N.F. (2021). Fall detection via inaudible acoustic sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 5.10.1145/3478094 ↗
25. Fu, Y., Wang, S., Zhong, L., Chen, L., Ren, J., and Zhang, Y. (2022, January 6–9). SVoice: Enabling Voice Communication in Silence via Acoustic Sensing on Commodity Devices. Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, Boston, MA, USA.10.1145/3560905.3568530 ↗
26. SymListener: Detecting Respiratory Symptoms via Acoustic Sensing in Driving Environments