Imaginary Speech Recognition Using a Convolutional Network with Long-Short Memory
Source abstract
Abstract
In recent years, a lot of researchers’ attentions were concentrating on imaginary speech understanding, decoding, and even recognition. Speech is a complex mechanism, which involves multiple brain areas in the process of production, planning, and precise control of a large number of muscles and articulation involved in the actual utterance. This paper proposes an intelligent imaginary speech recognition system of eleven different utterances, seven phonemes, and four words from the Kara One database. We showed, during our research, that the feature space of the cross-covariance in frequency domain offers a better perspective of the imaginary speech by computing LDA for 2D representation of the feature space, in comparison to cross-covariance in the time domain and the raw signals without any processing. In the classification stage, we used a CNNLSTM neural network and obtained a performance of 43% accuracy for all eleven different utterances. The developed system was meant to be a subject’s shared system. We also showed that, using the channels corresponding to the anatomical structures of the brain involved in speech production, i.e., Broca area, primary motor cortex, and secondary motor cortex, 93% of information is preserved, obtaining 40% accuracy by using 29 electrodes out of the initial 62.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: EEG and Brain-Computer Interfaces · Action Observation and Synchronization · Neuroscience and Music Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Applied Electronics and Information Engineering, Faculty of Electronics, Telecommunications and Information Technology, Polytechnic University of Bucharest, 060042 Bucharest, Romania
References (27)
Show source-provided references
1. Direct control of a computer from the human central nervous system10.1109/86.847815 ↗
2. Brain enabled mechanized speech synthesizer using Brain Mobile Interface
3. Spelling interface using intracortical signals in a completely locked-in patient enabled via auditory neurofeedback training10.1038/s41467-022-28859-8 ↗
4. Imagined speech can be decoded from low- and cross-frequency intracranial EEG features10.1038/s41467-021-27725-3 ↗
5. Word pair classification during imagined speech using direct brain recordings10.1038/srep25803 ↗
6. Speech synthesis from ECoG using densely connected 3D convolutional neural networks10.1088/1741-2552/ab0c59 ↗
7. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity10.1038/s42003-021-02578-0 ↗
8. Imagined Speech Classification Using Six Phonetically Distributed Words10.3389/frsip.2022.760643 ↗
9. Zhao, S., and Rudzicz, F. (2015, January 19–24). Classifying phonological categories in imagined and articulated speech. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7178118 ↗
10. Inferring imagined speech using EEG signals: A new approach using Riemannian manifold features10.1088/1741-2552/aa8235 ↗
11. Panachakel, J.T., Ramakrishnan, A.G., and Ananthapadmanabha, T.V. (2019, January 13–15). Decoding Imagined Speech using Wavelet Features and Deep Neural Networks. Proceedings of the 2019 IEEE 16th India Council International Conference (INDICON), Rajkot, India.10.1109/INDICON47234.2019.9028925 ↗
12. Saha, P., Fels, S., and Abdul-Mageed, M. (2019, January 13–17). Deep Learning the EEG Manifold for Phonological Categorization from Active Thoughts. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682330 ↗
13. A Long Short-Term Memory deep learning network for the prediction of epileptic seizures using EEG signals10.1016/j.compbiomed.2018.05.019 ↗
14. Electroencephalography-based imagined speech recognition using deep long short-term memory network10.4218/etrij.2021-0118 ↗
15. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2–4). Tensorflow: A system for large-scale machine learning. Proceedings of the 12th Symposium on Operating Systems Design and Implementation, Savannah, GA, USA.
16. Porbadnigk, A., Wester, M., and Calliess, T.S.J.P. (2009, January 14–17). EEG-based speech recognition—Impact of Temporal Effects. Proceedings of the International Conference on Bio-inspired Systems and Signal Processing, Porto, Portugal.
17. Rusnac, A.-L., and Grigore, O. (2022). CNN Architectures and Feature Extraction Methods for EEG Imaginary Speech Recognition. Sensors, 22.10.3390/s22134679 ↗
18. Bi-LSTM neural network for EEG-based error detection in musicians’ performance10.1016/j.bspc.2022.103885 ↗
19. A Novel Recurrent Neural Network to Classify EEG Signals for Customers’ Decision-Making Behavior Prediction in Brand Extension Scenario10.3389/fnhum.2021.610890 ↗
20. Classification of Hand Movements From EEG Using a Deep Attention-Based LSTM Network10.1109/JSEN.2019.2956998 ↗
21. Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W., and Woo, W. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. arXiv.
22. Scikit-learn: Machine Learning in Python
23. Brain regions that support accurate speech production after damage to Broca’s area10.1093/braincomms/fcab230 ↗
24. The Role of Primary Motor Cortex: More Than Movement Execution10.1080/00222895.2020.1738992 ↗
25. Secondary Motor Cortex Transforms Spatial Information into Planned Action during Navigation10.1016/j.cub.2020.03.016 ↗
26. Automated cortical projection of EEG sensors: Anatomical correlation via the international 10–10 system10.1016/j.neuroimage.2009.02.006 ↗
27. Notchenko, A., Kapushev, E., and Burnaev, E. (2017). Large-Scale Shape Retrieval with Sparse 3D Convolutional Neural Networks. arXiv.10.1007/978-3-319-73013-4_23 ↗