All-weather, natural silent speech recognition via machine-learning-assisted tattoo-like electronics
Source abstract
Abstract
Abstract The internal availability of silent speech serves as a translator for people with aphasia and keeps human–machine/human interactions working under various disturbances. This paper develops a silent speech strategy to achieve all-weather, natural interactions. The strategy requires few usage specialized skills like sign language but accurately transfers high-capacity information in complicated and changeable daily environments. In the strategy, the tattoo-like electronics imperceptibly attached on facial skin record high-quality bio-data of various silent speech, and the machine-learning algorithm deployed on the cloud recognizes accurately the silent speech and reduces the weight of the wireless acquisition module. A series of experiments show that the silent speech recognition system (SSRS) can enduringly comply with large deformation (~45%) of faces by virtue of the electricity-preferred tattoo-like electrodes and recognize up to 110 words covering daily vocabularies with a high average accuracy of 92.64% simply by use of small-sample machine learning. We successfully apply the SSRS to 1-day routine life, including daily greeting, running, dining, manipulating industrial robots in deafening noise, and expressing in darkness, which shows great promotion in real-world applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Sensor and Energy Harvesting Materials · Tactile and Sensory Interactions · Gaze Tracking and Assistive Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (51)
Show source-provided references
1. Wang, Y. H. et al. Electrically compensated, tattoo-like electrodes for epidermal electrophysiology at scale. Sci. Adv. 6, eabd0996 (2020).10.1126/sciadv.abd0996 ↗
2. Cai, S. et al. SVM-based classification of sEMG signals for upper-limb self-rehabilitation training. Front. Neurorobot. 13, 31 (2019).10.3389/fnbot.2019.00031 ↗
3. Cote-Allard, U. et al. Interpreting deep learning features for myoelectric control: a comparison with handcrafted features. Front. Bioeng. Biotechnol. 8, 158 (2020).10.3389/fbioe.2020.00158 ↗
4. Orjuela-Canon, A. D., Ruiz-Olaya, A. F. & Forero, L. Deep neural network for EMG signal classification of wrist position: Preliminary results. In 2017 IEEE Latin American Conference on Computational Intelligence (LA-CCI) 5 pp. (IEEE, 2017).
5. Jaramillo-Yánez, A., Benalcázar, M. E. & Mena-Maldonado, E. Real-time hand gesture recognition using surface electromyography and machine learning: a systematic literature review. Sensors 20, 2467 (2020).10.3390/s20092467 ↗
6. Jiang, Y. et al. Shoulder muscle activation pattern recognition based on sEMG and machine learning algorithms. Comput. Biol. Med. 197, 105721 (2020).
7. Sugie, N. & Tsunoda, K. A speech prosthesis employing a speech synthesizer - vowel discrimination from perioral muscle activities and vowel production. IEEE Trans. Biomed. Eng. 32, 485–490 (1985).10.1109/TBME.1985.325564 ↗
8. Morse, M. S. & Obrien, E. M. Research summary of a scheme to ascertain the availability of speech information in the myoelectric signals of neck and head muscles using surface electrodes. Comput. Biol. Med. 16, 399–410 (1986).10.1016/0010-4825(86)90064-8 ↗
9. Jorgensen, C., Lee, D. D. & Agabon, S. Sub Auditory Speech Recognition Based on EMG Signals. In International Joint Conference on Neural Networks 2003 3128–3133 (Institute of Electrical and Electronics Engineers Inc., 2003)10.1109/IJCNN.2003.1224072 ↗
10. Lee, K. S. EMG-based speech recognition using hidden Markov models with global control variables. IEEE Trans. Biomed. Eng. 55, 930–940 (2008).10.1109/TBME.2008.915658 ↗
11. Meltzner, G. S. et al. Development of sEMG sensors and algorithms for silent speech recognition. J. Neural Eng. 15, 046031 (2018).10.1088/1741-2552/aac965 ↗
12. Wang, Y. et al. Silent speech decoding using spectrogram features based on neuromuscular activities. Brain Sci. 10, 442 (2020).10.3390/brainsci10070442 ↗
13. Molina-Molina, A. et al. Validation of mDurance, a wearable surface electromyography system for muscle activity assessment. Front. Physiol. 11, 606287 (2020).10.3389/fphys.2020.606287 ↗
14. Peng, Y. H., Wang, X. J., Guo, L., Wang, Y. C. & Deng, Q. X. An efficient network coding-based fault-tolerant mechanism in WBAN for smart healthcare monitoring systems. Appl. Sci. 7, 18 (2017).10.3390/app7080817 ↗
15. Mehmood, G., Khan, M. Z., Abbas, S., Faisal, M. & Rahman, H. U. An energy-efficient and cooperative fault-tolerant communication approach for wireless body area network. IEEE Access 8, 69134–69147 (2020).10.1109/ACCESS.2020.2986268 ↗
16. Hsu, V. M., Wes, A. M., Tahiri, Y., Cornman-Homonoff, J. & Percec, I. Quantified facial soft-tissue strain in animation measured by real-time dynamic 3-dimensional imaging. Plast. Reconstr. Surg. Glob. Open 2, e211 (2014).10.1097/GOX.0000000000000185 ↗
17. Bracken, D. J., Ornelas, G., Coleman, T. P. & Weissbrod, P. A. High-density surface electromyography: a visualization method of laryngeal muscle activity. Laryngoscope 129, 2347–2353 (2019).10.1002/lary.27784 ↗
18. Kim, S. et al. Integrated wireless neural interface based on the Utah electrode array. Biomed. Microdevices 11, 453–466 (2009).10.1007/s10544-008-9251-y ↗
19. Liao, L. D., Wang, I. J., Chen, S. F., Chang, J. Y. & Lin, C. T. Design, fabrication and experimental validation of a novel dry-contact sensor for measuring electroencephalography signals without skin preparation. Sensors 11, 5819–5834 (2011).10.3390/s110605819 ↗
20. Kim, D. H. et al. Epidermal electronics. Science 333, 838–843 (2011).10.1126/science.1206157 ↗
21. Kim, Y. et al. A bioinspired flexible organic artificial afferent nerve. Science 360, 998–1003 (2018).10.1126/science.aao0098 ↗
22. Yang, J. C. et al. Electronic skin: recent progress and future prospects for skin-attachable devices for health monitoring, robotics, and prosthetics. Adv. Mater. 31, e1904765 (2019).10.1002/adma.201904765 ↗
23. Son, D. et al. An integrated self-healable electronic skin system fabricated via dynamic reconstruction of a nanostructured conducting network. Nat. Nanotechnol. 13, 1057–1065 (2018).10.1038/s41565-018-0244-6 ↗
24. Zhou, Y. et al. Multichannel noninvasive human–machine interface via stretchable µm thick sEMG patches for robot manipulation. J. Micromech. Microeng. 28, 014005 (2018).10.1088/1361-6439/aa9c2e ↗
25. Wang, Y. et al. Low-cost, μm-thick, tape-free electronic tattoo sensors with minimized motion and sweat artifacts. npj Flex. Electron. 2, 6 (2018).10.1038/s41528-017-0019-4 ↗
26. Liu, H. C. et al. An epidermal sEMG tattoo-like patch as a new human-machine interface for patients with loss of voice. Microsyst. Nanoeng. 6, 16 (2020).10.1038/s41378-019-0127-5 ↗
27. Widlund, T., Yang, S. X., Hsu, Y. Y. & Lu, N. S. Stretchability and compliance of freestanding serpentine-shaped ribbons. Int. J. Solids Struct. 51, 4026–4037 (2014).10.1016/j.ijsolstr.2014.07.025 ↗
28. Yang, X. et al. “Cut-and-paste” method for the rapid prototyping of soft electronics. Sci. China Technol. Sci. 62, 199–208 (2019).10.1007/s11431-018-9400-9 ↗
29. Vesa, E. P. & Ilie, B. Equipment for SEMG signals acquisition and processing. In International Conference on Advancements of Medicine and Health Care through Technology, MEDITECH 2014 187–192 (Springer Verlag, 2014).10.1007/978-3-319-07653-9_38 ↗
30. Ferreira, J. M. & Lima, C. Distributed system for acquisition and processing the sEMG signal. In 1st International Conference on Health Informatics, ICHI 2013 335–338 (Springer, 2013).10.1007/978-3-319-03005-0_85 ↗
31. Alemu, M., Kumar, D. K. & Bradley, A. Time-frequency analysis of SEMG−with special consideration to the interelectrode spacing. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 341–345 (2003).10.1109/TNSRE.2003.819903 ↗
32. Sheikh, M. U., Badihi, B., Ruttik, K. & Jantti, R. Adaptive physical layer selection for bluetooth 5: measurements and simulations. Wirel. Commun. Mob. Comput. 2021, 1–10 (2021).10.1155/2021/8842919 ↗
33. Sun, D. Z., Sun, L. & Yang, Y. On secure simple pairing in bluetooth standard v5.0-Part II: Privacy analysis and enhancement for low energy. Sensors 19, 3259 (2019).10.3390/s19153259 ↗
34. Lee, K. et al. Mechano-acoustic sensing of physiological processes and body motions via a soft wireless device placed at the suprasternal notch. Nat. Biomed. Eng. 4, 148–158 (2020).10.1038/s41551-019-0480-6 ↗
35. Koh, A. et al. A soft, wearable microfluidic device for the capture, storage, and colorimetric sensing of sweat. Sci. Transl. Med. 8, 366ra165 (2016).10.1126/scitranslmed.aaf2593 ↗
36. Hyoyoung, J. et al. NFC-enabled, tattoo-like stretchable biosensor manufactured by cut-and-paste method. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 4094–4097 (IEEE, 2017).
37. Tian, L. et al. Large-area MRI-compatible epidermal electronic interfaces for prosthetic control and cognitive monitoring. Nat. Biomed. Eng. 3, 194–205 (2019).10.1038/s41551-019-0347-x ↗
38. Huigen, E., Peper, A. & Grimbergen, C. A. Investigation into the origin of the noise of surface electrodes. Med. Biol. Eng. Comput. 40, 332–338 (2002).10.1007/BF02344216 ↗
39. Vicars, W. First 100 signs: American Sign Language (ASL). http://www.lifeprint.com/asl101/pages-layout/concepts.htm (2002).
40. Meltzner, G. S. et al. Speech recognition for vocalized and subvocal modes of production using surface EMG signals from the neck and face. In INTERSPEECH 2008 - 9th Annual Conference of the International Speech Communication Association 2667–2670 (International Speech Communication Association, 2008).10.21437/Interspeech.2008-661 ↗
41. Hooda, N., Das, R. & Kumar, N. Fusion of EEG and EMG signals for classification of unilateral foot movements. Biomed. Signal Process. Control 60, 101990 (2020).10.1016/j.bspc.2020.101990 ↗
42. Mithun, P., Pandey, P. C., Sebastian, T., Mishra, P. & Pandey, V. K. A wavelet based technique for suppression of EMG noise and motion artifact in ambulatory ECG. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2011, 7087–7090 (2011).
43. Xiao et al. Classification of surface EMG signal using relative wavelet packet energy. Comput Methods Prog. Biomed. 79, 189–195 (2005).10.1016/j.cmpb.2005.04.001 ↗
44. Pancholi, S. & Joshi, A. M. Electromyography-based hand gesture recognition system for upper limb amputees. Electron. Lett. 3, 1–4 (2019).
45. Yikang, Y. et al. A Multi-Gestures Recognition System Based on Less sEMG Sensors. In 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM) 105–110 (IEEE, 2019).10.1109/ICARM.2019.8834153 ↗
46. Too, J., Abdullah, A. R. & Saad, N. M. Classification of hand movements based on discrete wavelet transform and enhanced feature extraction. Int. J. Adv. Comput Sci. Appl. 10, 83–89 (2019).
47. Savur, C. & Sahin, F. Real-Time American Sign Language Recognition System Using Surface EMG Signal. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA) 497–502 (IEEE, 2015).10.1109/ICMLA.2015.212 ↗
48. Halder, S. et al. Online artifact removal for brain-computer interfaces using support vector machines and blind source separation. Comput. Intell. Neurosci. 2007, 82069–82069 (2007).10.1155/2007/82069 ↗
49. Sundaram, S. et al. Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 698–702 (2019).10.1038/s41586-019-1234-z ↗
50. Zhou, Z. et al. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nat. Electron. 3, 571–578 (2020).10.1038/s41928-020-0428-6 ↗
51. Luca, C. D. Surface electromyography: detection and recording. DelSys Incorporated 10, 1–10 (2002).