journal article·2025-02Open access

Decoding Silent Speech Cues From Muscular Biopotential Signals for Efficient Human‐Robot Collaborations

Penghao Dong, Sibo Tian, Si Chen, Yizong Li, Su Li, Minghui Zheng, Shanshan Yao

Advanced Materials Technologies· Volume 10· Issue 4

Abstract

Abstract Silent speech interfaces offer an alternative and efficient communication modality for individuals with voice disorders and when the vocalized speech communication is compromised by noisy environments. Despite the recent progress in developing silent speech interfaces, these systems face several challenges that prevent their wide acceptance, such as bulkiness, obtrusiveness, and immobility. Herein, the material optimization, structural design, deep learning algorithm, and system integration of mechanically and visually unobtrusive silent speech interfaces are presented that can realize both speaker identification and speech content identification. Conformal, transparent, and self‐adhesive electromyography electrode arrays are designed for capturing speech‐relevant muscle activities. Temporal convolutional networks are employed for recognizing speakers and converting sensing signals into spoken content. The resulting silent speech interfaces achieve a 97.5% speaker classification accuracy and 91.5% keyword classification accuracy using four electrodes. The speech interface is further integrated with an optical hand‐tracking system and a robotic manipulator for human‐robot collaboration in both assembly and disassembly processes. The integrated system achieves the control of the robot manipulator by silent speech and facilitates the hand‐over process by hand motion trajectory detection. The developed framework enables natural robot control in noisy environments and lays the ground for collaborative human‐robot tasks involving multiple human operators.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Muscle activation and electromyography studies · Robot Manipulation and Learning · Advanced Sensor and Energy Harvesting Materials

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Mechanical Engineering Stony Brook University Stony Brook NY 11794 USA

Department of Mechanical Engineering Texas A&M University College Station TX 77843 USA

Funding

National Science Foundation

References (90)

Show source-provided references
1. 10.3390/s2101002210.3390/s21010022 ↗
2. 10.1038/s41467-022-29083-010.1038/s41467-022-29083-0 ↗
3. 10.1088/1741-2552/aac96510.1088/1741-2552/aac965 ↗
4. 10.1145/355028110.1145/3550281 ↗
5. K.Sun C.Yu W.Shi L.Liu Y.Shi presented atProceedings of the 31st Annual ACM Symposium on User Interface Software and Technology ACM Berlin Germany2018 p.581.10.1145/3242587.3242599 ↗
6. Zhang Q. · Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. · 2021 · 5 · 43
7. G.Bolano A.Roennau R.Dillmann presented at27th IEEE International Symposium on Robot and Human Interactive Communication (RO‐MAN) IEEE Nanjing China2018 p.1075.10.1109/ROMAN.2018.8525671 ↗
8. G.Bolano L.Iviani A.Roennau R.Dillmann presented at2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO‐MAN) IEEE Vancouver Canada2021 p.806.10.1109/RO-MAN50785.2021.9515389 ↗
9. 10.1108/0143991051062924410.1108/01439910510629244 ↗
10. 10.1016/j.procir.2017.03.12610.1016/j.procir.2017.03.126 ↗
11. 10.2298/SJEE1301219T10.2298/SJEE1301219T ↗
12. 10.1016/j.engappai.2022.10559710.1016/j.engappai.2022.105597 ↗
13. 10.1109/TSMC.2020.300534010.1109/TSMC.2020.3005340 ↗
14. 10.1109/MRA.2018.281570410.1109/MRA.2018.2815704 ↗
15. 10.1016/S0007-8506(07)60026-210.1016/S0007-8506(07)60026-2 ↗
16. 10.1115/1.406399210.1115/1.4063992 ↗
17. 10.1109/TASE.2021.307487310.1109/TASE.2021.3074873 ↗
18. 10.1016/j.imavis.2014.06.00410.1016/j.imavis.2014.06.004 ↗
19. Y. M.Assael B.Shillingford S.Whiteson N.De Freitas (Preprint) arXiv arXiv:1611.015992016.
20. A.Fernandez‐Lopez O.Martinez F. M.Sukno presented atProceedings of the 2017 12th International Conference on Automatic Face and Gesture Recognition IEEE Washington DC USA2017 p.208.10.1109/FG.2017.34 ↗
21. 10.1109/TPAMI.2018.288905210.1109/TPAMI.2018.2889052 ↗
22. B.Denby M.Stone inProceedings of the 2004 IEEE International Conference on Acoustics Speech and Signal Processing IEEE Montreal Canada2004 p.685.
23. A.Jaumard‐Hakoun K.Xu C.Leboullenger P.Roussel‐Ragot B.Denby presented atProceedings of Interspeech 2016 ISCA San Francisco USA2016 p.1467.
24. 10.1038/s41593-023-01304-910.1038/s41593-023-01304-9 ↗
25. 10.1109/TMC.2016.251763010.1109/TMC.2016.2517630 ↗
26. 10.1109/MC.2015.31010.1109/MC.2015.310 ↗
27. 10.1016/j.specom.2012.02.00110.1016/j.specom.2012.02.001 ↗
28. 10.1016/j.specom.2010.01.00110.1016/j.specom.2010.01.001 ↗
29. 10.1109/TASLP.2017.275236510.1109/TASLP.2017.2752365 ↗
30. 10.1038/s41586-023-06443-410.1038/s41586-023-06443-4 ↗
31. 10.1038/s41586-023-06377-x10.1038/s41586-023-06377-x ↗
32. K.Brigham B. V.Kumar presented atProceedings of the 2010 4th International Conference on Bioinformatics and Biomedical Engineering IEEE Chengdu China2010 p.1.
33. 10.1073/pnas.94.26.1496510.1073/pnas.94.26.14965 ↗
34. P.Khanna T.Srivastava S.Pan S.Jain P.Nguyen presented atProceedings of the 22nd International Workshop on Mobile Computing Systems and Applications ACM Virtual United Kingdom2021 p.44.10.1145/3446382.3448363 ↗
35. J.Rekimoto Y.Nishimura presented atProceedings of Augmented Humans Conference 2021 ACM Rovaniemi Finland2021 p.91.10.1145/3458709.3458941 ↗
36. 10.1109/TASLP.2017.275726310.1109/TASLP.2017.2757263 ↗
37. 10.1016/0093-934X(87)90058-710.1016/0093-934X(87)90058-7 ↗
38. 10.1039/D3MH01062G10.1039/D3MH01062G ↗
39. Jin Y. · Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. · 2022 · 6 · 57
40. Wang J. · Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. · 2019 · 3 · 155
41. 10.1038/s41467-022-33457-910.1038/s41467-022-33457-9 ↗
42. 10.1021/acsami.2c1491810.1021/acsami.2c14918 ↗
43. 10.1038/s41528-021-00119-710.1038/s41528-021-00119-7 ↗
44. 10.1038/s41378-019-0127-510.1038/s41378-019-0127-5 ↗
45. 10.1002/smll.20220505810.1002/smll.202205058 ↗
46. 10.1016/j.ijheatmasstransfer.2022.12372010.1016/j.ijheatmasstransfer.2022.123720 ↗
47. 10.1016/j.cma.2022.11518310.1016/j.cma.2022.115183 ↗
48. 10.1016/j.cma.2021.11439410.1016/j.cma.2021.114394 ↗
49. 10.1038/s41528-023-00273-010.1038/s41528-023-00273-0 ↗
50. 10.1021/acsnano.0c0090610.1021/acsnano.0c00906 ↗
51. 10.1038/ncomms426610.1038/ncomms4266 ↗
52. 10.1002/admi.20220119710.1002/admi.202201197 ↗
53. 10.1002/adma.20180432710.1002/adma.201804327 ↗
54. 10.1038/s41427-020-00277-610.1038/s41427-020-00277-6 ↗
55. 10.1002/advs.20240421110.1002/advs.202404211 ↗
56. 10.1038/s41551-019-0480-610.1038/s41551-019-0480-6 ↗
57. 10.1038/s41467-020-18503-810.1038/s41467-020-18503-8 ↗
58. 10.1177/096032711986233310.1177/0960327119862333 ↗
59. 10.1021/la040134g10.1021/la040134g ↗
60. 10.1021/ie50254a01610.1021/ie50254a016 ↗
61. 10.3390/polym1211275210.3390/polym12112752 ↗
62. 10.1038/s41529-019-0080-710.1038/s41529-019-0080-7 ↗
63. 10.1002/aisy.20210003110.1002/aisy.202100031 ↗
64. 10.1021/acsnano.7b0218210.1021/acsnano.7b02182 ↗
65. OpenBCI Cyton Specs https://docs.openbci.com/Cyton/CytonSpecs/ (accessed: August 2024).
66. OpenBCI Cyton Getting Started Guidehttps://docs.openbci.com/GettingStarted/Boards/CytonGS/ (accessed: August 2024).
67. BrainFlow Python Real Time Plot https://brainflow.readthedocs.io/en/stable/Examples.html#python‐real‐time‐plot (accessed: August 2024).
68. 10.1109/JRPROC.1949.23296910.1109/JRPROC.1949.232969 ↗
69. NASA Visible Light https://science.nasa.gov/ems/09_visiblelight/ (accessed: April 2024).
70. 10.1016/j.matt.2022.08.00710.1016/j.matt.2022.08.007 ↗
71. 10.1038/nature1453910.1038/nature14539 ↗
72. 10.1016/j.taml.2023.10048510.1016/j.taml.2023.100485 ↗
73. C.Lea M. D.Flynn R.Vidal A.Reiter G. D.Hager presented atProceedings of the IEEE Conference on Computer Vision and Pattern Recognition IEEE Honolulu USA2017 p.156.
74. U.Hunkeler H. L.Truong A.Stanford‐Clark presented atProceedings of the 2008 3rd International Conference on Communication System Software and Middleware and Workshops IEEE Bangalore India2008 p.791.10.1109/COMSWA.2008.4554519 ↗
75. S.Tian X.Liang M. H.Zheng presented at2023 American Control Conference IEEE San Diego USA 2023 p.3356.10.23919/ACC55779.2023.10156342 ↗
76. 10.1109/LRA.2024.340111610.1109/LRA.2024.3401116 ↗
77. 10.1109/LRA.2024.335462810.1109/LRA.2024.3354628 ↗
78. Zhang X. · J. Comput. Inf. Sci. Eng. · 2024 · 24 · 5
79. 10.1002/celc.20150055510.1002/celc.201500555 ↗
80. 10.1016/j.snb.2016.10.00510.1016/j.snb.2016.10.005 ↗
81. 10.1007/s11837-016-1818-010.1007/s11837-016-1818-0 ↗
82. 10.1021/acsami.9b0752010.1021/acsami.9b07520 ↗
83. 10.1109/ACCESS.2019.289759010.1109/ACCESS.2019.2897590 ↗
84. 10.1002/admt.20210163710.1002/admt.202101637 ↗
85. 10.1002/adma.20130192110.1002/adma.201301921 ↗
86. 10.1115/1.400596310.1115/1.4005963 ↗
87. 10.1098/rsif.2011.058310.1098/rsif.2011.0583 ↗
88. 10.1039/D2CS00652A10.1039/D2CS00652A ↗
89. 10.1002/adma.20200372310.1002/adma.202003723 ↗
90. Maaten L. V. D. · J. Mach. Learn. Res. · 2008 · 9 · 2579